venkata91 opened a new pull request, #18947:
URL: https://github.com/apache/iceberg/pull/18947
## What Changed
- `ParquetIO.file(OutputFile)` now wraps a `HadoopOutputFile` in a new
`HadoopFileSystemOutputFile` that creates the stream through the file's own
`FileSystem`. Before, it used Parquet's `HadoopOutputFile.fromPath`, which
looks the `FileSystem` up again from the path and Hadoop `Configuration`, so it
ignored the instance the Iceberg file was created with.
- Removed the `ParquetIO.file(OutputFile, ParquetConfiguration)` overload.
`ParquetWriter` no longer keeps the `ParquetConfiguration` field and calls
`ParquetIO.file(output)` instead, so a Hadoop `Configuration` is no longer
built from the Parquet write config when opening the output stream.
- Added `TestParquet.writeUsesFileSystemOfHadoopOutputFile`. It wraps the
local `FileSystem` in a counting `FilterFileSystem`, checks that the write
calls `create` exactly once on that `FileSystem`, and checks that the file
reads back with one record.
## Risk Assessment
✅ Low: The change is small and package-private. I checked its create,
block-size and path semantics against parquet-hadoop's HadoopOutputFile
bytecode (1.18.1 jar, the closest one cached; the build pins 1.18.2), and
nothing in ParquetFileWriter depends on parquet's own HadoopOutputFile type, so
there is no API change and default-config behavior is unchanged.
## Testing
I ran the real Iceberg Parquet writer and reader against a local Hadoop
filesystem, on both the target commit and the base. With the fix, a write
through a HadoopOutputFile built on a counting FilterFileSystem calls that
filesystem's create exactly once and the file reads back. With the base's main
code the same test fails (creates == 0). Writing through HadoopFileIO
round-trips correctly. A plain create on an existing file is rejected and
leaves the original rows intact, while overwrite() replaces them. The new
output file reports the same path, supportsBlockSize and defaultBlockSize as
parquet-hadoop's own HadoopOutputFile. All of TestParquet and
TestParquetEncryption pass. The scratch test was removed from the worktree; a
copy and the logs are in the evidence directory.
- Live validation: ✅ go - 6 of 6 scenarios driven live against the product
| Scenario | Result | Live | Evidence |
| --- | --- | --- | --- |
| Parquet write via HadoopOutputFile built with a custom (counting)
FileSystem on a local temp file calls that FS's create once and the file is
readable | ✅ pass | live | TestParquet.writeUsesFileSystemOfHadoopOutputFile
passed on target (target-tests.log, TEST-...TestParquet.xml) |
| Adversarial: same write with base-commit ParquetIO bypasses the provided
FileSystem | ✅ pass | live | base-regression-test.log: expected 1 but was 0 |
| HadoopFileIO local write/read round trip still works (3 rows written and
read back) | ✅ pass | live | Scratch output 'NM roundtrip read: [1, 2,
3]' |
| Adversarial: create without overwrite on an existing file fails and leaves
original data; overwrite() replaces it | ✅ pass | live | Scratch output
'create-on-existing threw: UncheckedIOException' then 'after
overwrite read: [7, 8]' |
| New output file reports the same path, supportsBlockSize and
defaultBlockSize as parquet-hadoop's HadoopOutputFile for the same local
path/conf | ✅ pass | live | Scratch output lines 'NM ours' vs 'NM
parquet-hadoop' |
| Encrypted Parquet writes/reads still work (TestParquetEncryption) and
existing TestParquet suite passes | ✅ pass | live | TestParquetEncryption 3/3,
TestParquet 37/37 in target-tests.log |
<details>
<summary>Evidence: Target run log (TestParquet, TestParquetEncryption,
scratch scenarios)</summary>
```text
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0 9 43764 9 4125 0 0 19764 0 0:00:02 --:--:-- 0:00:02
19736 100 43764 100 43764 0 0 191k 0 --:--:-- --:--:-- --:--:--
191k
Initialized native services in: ~/.gradle/native
Initialized jansi services in: ~/.gradle/native
The client will now receive all logging from the daemon (pid: 2911512). The
daemon log file: ~/.gradle/daemon/8.14.6/daemon-2911512.out.log
Starting 8th build in daemon [uptime: 7 mins 16.104 secs, performance: 100%,
GC rate: 0.00/s, heap usage: 0% of 1.5 GiB]
Using 15 worker leases.
Configuration on demand is an incubating feature.
Not watching {run-worktree} since the file system is not supported
Watching the file system is configured to be enabled if available
File system watching is active
Starting Build
Settings evaluated using settings file '{run-worktree}/settings.gradle'.
Using local directory build cache for the root build (location =
~/.gradle/caches/build-cache-1, remove unused entries = after 7 days).
Projects loaded. Root project using build file '{run-worktree}/build.gradle'.
Included projects: [root project 'iceberg', project ':iceberg-aliyun',
project ':iceberg-api', project ':iceberg-arrow', project ':iceberg-aws',
project ':iceberg-aws-bundle', project ':iceberg-azure', project
':iceberg-azure-bundle', project ':iceberg-bigquery', project ':iceberg-bom',
project ':iceberg-bundled-guava', project ':iceberg-common', project
':iceberg-core', project ':iceberg-data', project ':iceberg-dell', project
':iceberg-delta-lake', project ':iceberg-flink', project ':iceberg-gcp',
project ':iceberg-gcp-bundle', project ':iceberg-hive-metastore', project
':iceberg-kafka-connect', project ':iceberg-mr', project ':iceberg-nessie',
project ':iceberg-open-api', project ':iceberg-orc', project
':iceberg-parquet', project ':iceberg-snowflake', project ':iceberg-spark',
project ':iceberg-flink:iceberg-flink-1.20', project
':iceberg-flink:iceberg-flink-2.2', project ':iceberg-flink:iceberg-flink-2.3',
project ':iceberg-flink:iceberg-flink-runtime-1.20', project ':iceberg
-flink:iceberg-flink-runtime-2.2', project
':iceberg-flink:iceberg-flink-runtime-2.3', project
':iceberg-kafka-connect:iceberg-kafka-connect', project
':iceberg-kafka-connect:iceberg-kafka-connect-events', project
':iceberg-kafka-connect:iceberg-kafka-connect-runtime', project
':iceberg-kafka-connect:iceberg-kafka-connect-transforms', project
':iceberg-spark:iceberg-spark-4.2_2.13', project
':iceberg-spark:iceberg-spark-extensions-4.2_2.13', project
':iceberg-spark:iceberg-spark-runtime-4.2_2.13']
Task path ':iceberg-parquet:test' matched project ':iceberg-parquet'
> Configure project :
Evaluating root project 'iceberg' using build file
'{run-worktree}/build.gradle'.
Starting process 'command 'git''. Working directory: {run-worktree} Command:
git describe --tags --always --first-parent --abbrev=7 --match=apache-iceberg-*
HEAD
Successfully started process 'command 'git''
Starting process 'command 'git''. Working directory: {run-worktree} Command:
git status --porcelain
Successfully started process 'command 'git''
Generating iceberg-build.properties from git
Starting process 'command 'git''. Working directory: {run-worktree} Command:
git branch --show-current
Successfully started process 'command 'git''
> Configure project :iceberg-parquet
Evaluating project ':iceberg-parquet' using build file
'{run-worktree}/parquet/build.gradle'.
Task name matched 'test'
Selected primary task 'test' from project :iceberg-parquet
> Configure project :iceberg-api
Evaluating project ':iceberg-api' using build file
'{run-worktree}/api/build.gradle'.
> Configure project :iceberg-bundled-guava
Evaluating project ':iceberg-bundled-guava' using build file
'{run-worktree}/bundled-guava/build.gradle'.
> Configure project :iceberg-core
Evaluating project ':iceberg-core' using build file
'{run-worktree}/core/build.gradle'.
> Configure project :iceberg-common
Evaluating project ':iceberg-common' using build file
'{run-worktree}/common/build.gradle'.
All projects evaluated.
Tasks to be executed: [task ':generateGitProperties', task ':buildInfo',
task ':iceberg-bundled-guava:compileJava', task
':iceberg-bundled-guava:processResources', task
':iceberg-bundled-guava:classes', task
':iceberg-bundled-guava:compileTestJava', task
':iceberg-bundled-guava:shadowJar', task ':iceberg-api:compileJava', task
':iceberg-common:compileJava', task ':iceberg-core:compileJava', task
':iceberg-parquet:compileJava', task ':iceberg-parquet:processResources', task
':iceberg-parquet:classes', task ':iceberg-parquet:jar', task
':iceberg-parquet:compileTestFixturesJava', task
':iceberg-parquet:processTestFixturesResources', task
':iceberg-parquet:testFixturesClasses', task
':iceberg-parquet:testFixturesJar', task ':iceberg-api:processResources', task
':iceberg-api:classes', task ':iceberg-api:jar', task
':iceberg-api:compileTestJava', task ':iceberg-api:processTestResources', task
':iceberg-api:testClasses', task ':iceberg-api:testJar', task
':iceberg-common:processResources
', task ':iceberg-common:classes', task ':iceberg-common:jar', task
':iceberg-core:processResources', task ':iceberg-core:classes', task
':iceberg-core:jar', task ':iceberg-core:compileTestJava', task
':iceberg-core:processTestResources', task ':iceberg-core:testClasses', task
':iceberg-core:testJar', task ':iceberg-parquet:compileTestJava', task
':iceberg-parquet:processTestResources', task ':iceberg-parquet:testClasses',
task ':iceberg-parquet:test']
Tasks that were excluded: []
Resolve mutations for :generateGitProperties (Thread[Execution
worker,5,main]) started.
Resolve mutations for :iceberg-bundled-guava:compileJava (Thread[Execution
worker Thread 2,5,main]) started.
Resolve mutations for :iceberg-parquet:processResources (Thread[Execution
worker Thread 3,5,main]) started.
Resolve mutations for :iceberg-api:processResources (Thread[Execution worker
Thread 4,5,main]) started.
Resolve mutations for :iceberg-common:processResources (Thread[Execution
worker Thread 5,5,main]) started.
:generateGitProperties (Thread[Execution worker,5,main]) started.
:iceberg-parquet:processResources (Thread[Execution worker Thread 3,5,main])
started.
:iceberg-api:processResources (Thread[Execution worker Thread 6,5,main])
started.
:iceberg-common:processResources (Thread[Execution worker Thread 5,5,main])
started.
Resolve mutations for :iceberg-core:processResources (Thread[Execution
worker Thread 4,5,main]) started.
:iceberg-bundled-guava:compileJava (Thread[Execution worker Thread
2,5,main]) started.
:iceberg-core:processResources (Thread[Execution worker Thread 8,5,main])
started.
> Task :iceberg-parquet:processResources NO-SOURCE
Skipping task ':iceberg-parquet:processResources' as it has no source files
and no previous output files.
> Task :iceberg-api:processResources NO-SOURCE
Skipping task ':iceberg-api:processResources' as it has no source files and
no previous output files.
> Task :iceberg-common:processResources NO-SOURCE
Skipping task ':iceberg-common:processResources' as it has no source files
and no previous output files.
Resolve mutations for :iceberg-parquet:processTestFixturesResources
(Thread[Execution worker Thread 3,5,
... [28134 bytes truncated] ...
apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.gz]
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.zstd]
TestParquet > testRowGroupSizeConfigurableWithWriter() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > perColumnCompressionCodecForNonExistentColumnIsIgnored()
STANDARD_ERROR
[Test worker] WARN org.apache.iceberg.parquet.Parquet - Skipping
per-column compression config for missing field: non_existent_field
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
TestParquet > testPerColumnDictionaryEncodingForNonExistentColumnIsIgnored()
STANDARD_ERROR
[Test worker] WARN org.apache.iceberg.parquet.Parquet - Skipping
dictionary encoding config for missing field: non_existent_field
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > adaptiveBloomFilterSizingShrinksFile() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] INFO
org.apache.parquet.column.values.bloomfilter.AdaptiveBlockSplitBloomFilter -
The number of distinct values in [id] is approximately 5, the optimal bloom
filter can accept 216500 distinct values, byte size is 262144.
TestParquet > deleteRowGroupSizeLargerThanIntegerMax() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testMetricsMissingColumnStatisticsInRowGroups() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testPerColumnDictionaryEncoding() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testRowGroupSizeConfigurable() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] WARN org.apache.parquet.hadoop.InternalParquetRecordWriter
- Too much memory used: Store {
[intCol] optional int32 intCol = 1 {
r:0 bytes
d:0 bytes
data: FallbackValuesWriter{
data: initial: DictionaryValuesWriter{
data: initial: dict:400
data: initial: values:400
data: initial:}
data: fallback: PLAIN CapacityByteArrayOutputStream 0 slabs, 0 bytes
data:}
pages: ColumnChunkPageWriter ConcatenatingByteBufferCollector 0
slabs, 0 bytes
total: 400/800
}
}
TestParquet > booleanNotEqualFilter() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] INFO org.apache.parquet.filter2.compat.FilterCompat -
Filtering using predicate: noteq(flag, true)
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- RecordReader initialized will read a total of 3 records.
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- at row 0. reading next block
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- block read in memory in 0 ms. row count = 3
TestParquet > testTwoLevelList() STANDARD_ERROR
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- RecordReader initialized will read a total of 1 records.
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- at row 0. reading next block
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- block read in memory in 0 ms. row count = 1
TestParquet > testColumnStatisticsEnabled() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > rowGroupSizeLargerThanIntegerMax() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testPerColumnDictionaryEncodingViaBuilder() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testPerColumnDictionaryEncodingNestedField() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > testGeospatialFooterMetricsSkipParquetBounds() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > perColumnCompressionCodec() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.zstd]
TestParquet > testNumberOfBytesWritten() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquet > timestampNanoFilterRespectsNanoseconds() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] INFO org.apache.parquet.filter2.compat.FilterCompat -
Filtering using predicate: gteq(ts, 1704067200000000500)
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- RecordReader initialized will read a total of 5 records.
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- at row 0. reading next block
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- block read in memory in 0 ms. row count = 5
TestParquetEncryption > testReadEncryptedFileWithoutAADPrefix()
STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquetEncryption > testReadEncryptedFileWithoutKeys() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
TestParquetEncryption > testReadEncryptedFile() STANDARD_ERROR
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.gz]
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- RecordReader initialized will read a total of 100 records.
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- at row 0. reading next block
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new decompressor [.gz]
[Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader
- block read in memory in 0 ms. row count = 100
Gradle Test Executor 16 finished executing tests.
> Task :iceberg-parquet:test
Finished generating test XML results (0.001 secs) into:
{run-worktree}/parquet/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.003 secs) into:
{run-worktree}/parquet/build/reports/tests/test
Stored cache entry for task ':iceberg-parquet:test' with cache key
42035343ff53f52dda8bf660815f9885
[Incubating] Problems report is available at:
file://{run-worktree}/build/reports/problems/problems-report.html
Deprecated Gradle features were used in this build, making it incompatible
with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
For more on this, please refer to
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
in the Gradle documentation.
BUILD SUCCESSFUL in 13s
22 actionable tasks: 13 executed, 9 from cache
Some of the file system contents retained in the virtual file system are on
file systems that Gradle doesn't support watching. The relevant state was
discarded to ensure changes to these locations are properly detected. You can
override this by explicitly enabling file system watching.
```
</details>
<details>
<summary>Evidence: Base regression run (test fails on base: expected 1 but
was 0)</summary>
```text
Configuration on demand is an incubating feature.
> Task :iceberg-parquet:processResources NO-SOURCE
> Task :iceberg-api:processResources NO-SOURCE
> Task :iceberg-core:processResources NO-SOURCE
> Task :iceberg-common:processResources NO-SOURCE
> Task :iceberg-core:processTestResources UP-TO-DATE
> Task :iceberg-parquet:processTestFixturesResources UP-TO-DATE
> Task :iceberg-parquet:processTestResources NO-SOURCE
> Task :generateGitProperties
> Task :buildInfo
> Task :iceberg-api:processTestResources
> Task :iceberg-bundled-guava:compileJava UP-TO-DATE
> Task :iceberg-bundled-guava:processResources NO-SOURCE
> Task :iceberg-bundled-guava:classes UP-TO-DATE
> Task :iceberg-bundled-guava:compileTestJava NO-SOURCE
> Task :iceberg-bundled-guava:shadowJar UP-TO-DATE
> Task :iceberg-common:compileJava UP-TO-DATE
> Task :iceberg-common:classes UP-TO-DATE
> Task :iceberg-common:jar UP-TO-DATE
> Task :iceberg-api:compileJava UP-TO-DATE
> Task :iceberg-api:classes UP-TO-DATE
> Task :iceberg-api:jar UP-TO-DATE
> Task :iceberg-api:compileTestJava UP-TO-DATE
> Task :iceberg-api:testClasses
> Task :iceberg-core:compileJava UP-TO-DATE
> Task :iceberg-core:classes UP-TO-DATE
> Task :iceberg-core:jar UP-TO-DATE
> Task :iceberg-api:testJar
file '{run-worktree}/build/iceberg-build.properties' will be copied to
'iceberg-build.properties', overwriting file
'{run-worktree}/api/build/resources/test/iceberg-build.properties', which has
already been copied there.
> Task :iceberg-parquet:compileJava FROM-CACHE
> Task :iceberg-parquet:classes UP-TO-DATE
> Task :iceberg-core:compileTestJava UP-TO-DATE
> Task :iceberg-core:testClasses UP-TO-DATE
> Task :iceberg-core:testJar UP-TO-DATE
> Task :iceberg-parquet:jar
> Task :iceberg-parquet:compileTestFixturesJava FROM-CACHE
> Task :iceberg-parquet:testFixturesClasses UP-TO-DATE
> Task :iceberg-parquet:testFixturesJar UP-TO-DATE
> Task :iceberg-parquet:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :iceberg-parquet:testClasses
> Task :iceberg-parquet:test FAILED
TestParquet > writeUsesFileSystemOfHadoopOutputFile() FAILED
org.opentest4j.AssertionFailedError:
expected: 1
but was: 0
at
app//org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile(TestParquet.java:1099)
Failures:
org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile()
1 test completed, 1 failed
[Incubating] Problems report is available at:
file://{run-worktree}/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':iceberg-parquet:test'.
> There were failing tests. See the report at:
file://{run-worktree}/parquet/build/reports/tests/test/index.html
* Try:
> Run with --scan to get full insights.
Deprecated Gradle features were used in this build, making it incompatible
with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
For more on this, please refer to
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
in the Gradle documentation.
BUILD FAILED in 7s
22 actionable tasks: 6 executed, 2 from cache, 14 up-to-date
```
</details>
<details>
<summary>Evidence: Scratch scenario test source</summary>
```text
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iceberg.parquet;
import static org.apache.iceberg.types.Types.NestedField.required;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.File;
import java.nio.file.Path;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.iceberg.Schema;
import org.apache.iceberg.data.GenericRecord;
import org.apache.iceberg.data.Record;
import org.apache.iceberg.data.parquet.GenericParquetWriter;
import org.apache.iceberg.data.parquet.InternalReader;
import org.apache.iceberg.hadoop.HadoopFileIO;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.io.FileAppender;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Types;
import org.apache.parquet.io.OutputFile;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
class TestNmScratchWrite {
private static final Schema SCHEMA = new Schema(required(1, "id",
Types.LongType.get()));
@TempDir Path temp;
private void write(HadoopFileIO io, String loc, boolean overwrite, long...
ids) throws Exception {
Parquet.WriteBuilder b =
Parquet.write(io.newOutputFile(loc)).schema(SCHEMA).createWriterFunc(GenericParquetWriter::create);
if (overwrite) {
b = b.overwrite();
}
try (FileAppender<Record> a = b.build()) {
for (long id : ids) {
Record r = GenericRecord.create(SCHEMA);
r.setField("id", id);
a.add(r);
}
}
}
private List<Long> read(HadoopFileIO io, String loc) throws Exception {
List<Long> out = Lists.newArrayList();
try (CloseableIterable<Record> it =
Parquet.read(io.newInputFile(loc)).project(SCHEMA)
.createReaderFunc(s -> InternalReader.create(SCHEMA,
s)).build()) {
for (Record r : it) {
out.add((Long) r.get(0));
}
}
return out;
}
@Test
void hadoopFileIORoundTripCreateAndOverwrite() throws Exception {
HadoopFileIO io = new HadoopFileIO(new Configuration());
String loc = new File(temp.toFile(), "a.parquet").toURI().toString();
write(io, loc, false, 1L, 2L, 3L);
System.out.println("NM roundtrip read: " + read(io, loc));
assertThat(read(io, loc)).containsExactly(1L, 2L, 3L);
assertThatThrownBy(() -> write(io, loc, false, 9L))
.satisfies(e -> System.out.println("NM create-on-existing threw: " +
e));
assertThat(read(io, loc)).containsExactly(1L, 2L, 3L);
write(io, loc, true, 7L, 8L);
System.out.println("NM after overwrite read: " + read(io, loc));
assertThat(read(io, loc)).containsExactly(7L, 8L);
}
@Test
void outputFileMetadataMatchesParquetHadoopOutputFile() throws Exception {
Configuration conf = new Configuration();
String loc = new File(temp.toFile(), "b.parquet").getAbsolutePath();
OutputFile ours = ParquetIO.file(new
HadoopFileIO(conf).newOutputFile(loc));
OutputFile theirs =
org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(new
org.apache.hadoop.fs.Path(loc), conf);
System.out.println("NM ours: path=" + ours.getPath() + "
supportsBlockSize=" + ours.supportsBlockSize()
+ " defaultBlockSize=" + ours.defaultBlockSize() + " class=" +
ours.getClass().getName());
System.out.println("NM parquet-hadoop: path=" + theirs.getPath() + "
supportsBlockSize="
+ theirs.supportsBlockSize() + " defaultBlockSize=" +
theirs.defaultBlockSize());
assertThat(ours.getPath()).isEqualTo(theirs.getPath());
assertThat(ours.supportsBlockSize()).isEqualTo(theirs.supportsBlockSize());
assertThat(ours.defaultBlockSize()).isEqualTo(theirs.defaultBlockSize());
}
}
```
</details>
<details>
<summary>Evidence: Scratch scenario output</summary>
```text
NM ours: path=file:/tmp/junit-.../b.parquet supportsBlockSize=false
defaultBlockSize=33554432
class=org.apache.iceberg.parquet.ParquetIO$HadoopFileSystemOutputFile
NM parquet-hadoop: path=file:/tmp/junit-.../b.parquet
supportsBlockSize=false defaultBlockSize=33554432
NM roundtrip read: [1, 2, 3]
NM create-on-existing threw: java.io.UncheckedIOException: Failed to create
Parquet file
NM after overwrite read: [7, 8]
```
</details>
- Evidence: TestParquet JUnit XML (local file:
<code>~/.no-mistakes/evidence/01M7EBHZ3M1YRZ9N4PS8EKC29F/TEST-org.apache.iceberg.parquet.TestParquet.xml</code>)
## Pipeline
Updates from [git push
no-mistakes](https://github.com/kunchenguid/no-mistakes)
<!-- no-mistakes-pipeline-attestation:v1
{"head_sha":"d4de555cf518fef893bd609aefb66eb53c9b1247","steps":[{"step":"intent","status":"completed"},{"step":"rebase","status":"skipped"},{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"},{"step":"lint","status":"completed"},{"step":"push","status":"completed"},{"step":"pr","status":"running"},{"step":"ci","status":"pending"}],"live_validation":{"verdict":"go","live":6,"total":6}}
-->
<details>
<summary>✅ **Intent** - passed</summary>
✅ No issues found.
</details>
<details>
<summary>⏭️ **Rebase** - skipped</summary>
Step was skipped.
</details>
<details>
<summary>⚠️ **Review** - 1 info</summary>
- ℹ️
`parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java:118` -
FileSystem resolution for writes no longer sees writer-level config. The old
ParquetIO.file(output, conf) built a Hadoop Configuration from the writer's
ParquetConfiguration, which is the HadoopOutputFile conf plus every
WriteBuilder.set/setAll entry, including table properties (Parquet.java:193,
514-516), and called FileSystem.get with it. The new code uses
hfile.getFileSystem(), resolved from the file's own conf. The two only
differ when the FS cache is disabled for the scheme
(fs.<scheme>.impl.disable.cache=true) and FS settings are passed as table
or writer properties. In that case the old code created a fresh, never-closed
FileSystem per data file with those overrides applied; the new code reuses the
FileIO's FS. This fits the author's stated design and arguably fixes a
leak, so no action is needed. It is noted only for the PR discussion.
</details>
<details>
<summary>✅ **Test** - passed</summary>
✅ No issues found.
- Live validation: ✅ go - 6 of 6 scenarios driven live against the product
| Scenario | Result | Live | Evidence |
| --- | --- | --- | --- |
| Parquet write via HadoopOutputFile built with a custom (counting)
FileSystem on a local temp file calls that FS's create once and the file is
readable | ✅ pass | live | TestParquet.writeUsesFileSystemOfHadoopOutputFile
passed on target (target-tests.log, TEST-...TestParquet.xml) |
| Adversarial: same write with base-commit ParquetIO bypasses the provided
FileSystem | ✅ pass | live | base-regression-test.log: expected 1 but was 0 |
| HadoopFileIO local write/read round trip still works (3 rows written and
read back) | ✅ pass | live | Scratch output 'NM roundtrip read: [1, 2,
3]' |
| Adversarial: create without overwrite on an existing file fails and leaves
original data; overwrite() replaces it | ✅ pass | live | Scratch output
'create-on-existing threw: UncheckedIOException' then 'after
overwrite read: [7, 8]' |
| New output file reports the same path, supportsBlockSize and
defaultBlockSize as parquet-hadoop's HadoopOutputFile for the same local
path/conf | ✅ pass | live | Scratch output lines 'NM ours' vs 'NM
parquet-hadoop' |
| Encrypted Parquet writes/reads still work (TestParquetEncryption) and
existing TestParquet suite passes | ✅ pass | live | TestParquetEncryption 3/3,
TestParquet 37/37 in target-tests.log |
- <code>`./gradlew :iceberg-parquet:test --tests
org.apache.iceberg.parquet.TestParquet --tests
org.apache.iceberg.parquet.TestParquetEncryption --tests
org.apache.iceberg.parquet.TestNmScratchWrite --offline` on target commit (37 +
3 + 2 tests, 0 failures)</code>
- <code>`./gradlew :iceberg-parquet:test --tests
org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile`
with parquet/src/main reverted to base 234d55774 (fails: expected 1 but was
0)</code>
- `Temporary scratch test (removed afterwards): HadoopFileIO local
write/read round trip, create on existing file rejected, overwrite() replaces
contents`
- `Temporary scratch test: ParquetIO.file(HadoopFileIO output)
getPath/supportsBlockSize/defaultBlockSize compared against parquet-hadoop
HadoopOutputFile.fromPath for the same path and conf`
</details>
<details>
<summary>✅ **Document** - passed</summary>
✅ No issues found.
</details>
<details>
<summary>✅ **Lint** - passed</summary>
✅ No issues found.
</details>
<details>
<summary>✅ **Push** - passed</summary>
✅ No issues found.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]