Tishj opened a new issue, #17535:
URL: https://github.com/apache/iceberg/issues/17535
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Spark
```sql
CREATE OR REPLACE TABLE iceberg_catalog.test_add_files.fake_data_test (
id bigint,
name string,
category string,
value double,
count bigint,
active boolean,
description string
)
USING iceberg
TBLPROPERTIES (
'format-version'='2'
)
CALL iceberg_catalog.system.add_files(
table => 'test_add_files.fake_data_test',
source_table => 'parquet.`{FILES_DIR}`'
)
```
This produces the Avro key-value metadata:
```
┌───────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ key │
value
│
│ varchar │
varchar
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ schema │
{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"name","required":false,"type":"st
│
│ │
ring"},{"id":3,"name":"category","required":false,"type":"string"},{"id":4,"name":"value","required":false,"type":"double"},{"id":5,"name":"co
│
│ │
unt","required":false,"type":"long"},{"id":6,"name":"active","required":false,"type":"boolean"},{"id":7,"name":"description","required":false,
│
│ │ "type":"string"}]}
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ avro.schema │
{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"d
│
│ │
efault":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Locati
│
│ │ on URI with FS
scheme","field-id":100},{"name":"file_format","type":"string","doc":"File
format name: avro, orc, or parquet","field-id":101},{ │
│ │
"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number
of re │
│ │ cords in the
file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total
file size in bytes","field-id":104},{"name":"block_ │
│ │
size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118…
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ avro.codec │ deflate
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ format-version │ 1
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ partition-spec-id │ 0
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ iceberg.schema │
{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"
│
│ │
type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type"
│
│ │ :"string","doc":"Location URI with FS
scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File
format name: avro, o │
│ │ rc, or
parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required"
│
│ │ :true,"type":"long","doc":"Number of records in the
file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total
fi │
│ │ le size in
bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"ty…
│
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ partition-spec │ []
│
└───────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
That `format_version` reports `1`, but the actual manifest file's schema
suggests v2 (as does the version of the table)
Most notably, the `manifest_entry.data_file.snapshot_id` attribute is
missing, which is fine for v2, but not for v1, where it's marked as `required`.
If a reader trusts the `format_version` obtained from the manifest file's
Avro key-value metadata, the reader will error.
And the contents of the file (not relevant, but sharing for completeness)
```
┌────────┬─────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ status │ snapshot_id │
data_file
│
│ int32 │ int64 │ struct(file_path varchar, file_format varchar,
"partition" "null", record_count bigint, file_size_in_bytes bigint,
block_size_in_bytes bigi │
│ │ │ nt, column_sizes map(integer, bigint), value_counts
map(integer, bigint), null_value_counts map(integer, bigint), nan_value_counts
map(inte │
│ │ │ ger, bigint), lower_bounds map(integer, blob),
upper_bounds map(integer, blob), key_metadata blob, split_offsets bigint[],
sort_order_id in │
│ │ │
teger)
│
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ {
│
│ │ │ 'file_path': 'file:/output/files/file_0.parquet',
│
│ │ │ 'file_format': PARQUET, 'partition': null,
'record_count': 22402, 'file_size_in_bytes': 952597, 'block_size_in_bytes':
67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {},
'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {},
'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null,
'sort_order_id': 0
│
│ │ │ }
│
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ {
│
│ │ │ 'file_path': 'file:/output/files/file_1.parquet',
│
│ │ │ 'file_format': PARQUET, 'partition': null,
'record_count': 33436, 'file_size_in_bytes': 1418456, 'block_size_in_bytes':
67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {},
'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {},
'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null,
'sort_order_id': 0
│
│ │ │ }
│
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ {
│
│ │ │ 'file_path': 'file:/output/files/file_2.parquet',
│
│ │ │ 'file_format': PARQUET, 'partition': null,
'record_count': 44162, 'file_size_in_bytes': 1875070, 'block_size_in_bytes':
67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {},
'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {},
'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null,
'sort_order_id': 0
│
│ │ │ }
│
└────────┴─────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Not super relevant, but the parquet files were created with DuckDB:
```py
con = duckdb.connect()
con.execute("select setseed(0.42)")
# Generate synthetic data inside DuckDB
con.execute(f"""
CREATE OR REPLACE TABLE fake_data AS
SELECT
i AS id,
'item_' || i AS name,
CASE floor(random() * 3)
WHEN 0 THEN 'A'
WHEN 1 THEN 'B'
ELSE 'C'
END AS category,
100 + 15 * random() AS value,
CAST(floor(random() * 99) + 1 AS INTEGER) AS count,
random() < 0.5 AS active,
substr(md5(random()::VARCHAR), 1, 24) AS description
FROM range(1, {n_rows + 1}) t(i)
""")
# Get categories
categories = [row[0] for row in con.execute(
"SELECT DISTINCT category FROM fake_data"
).fetchall()]
# Write one parquet file per category
for i, category in enumerate(categories):
file_path = parquet_dir / f"file_{i}.parquet"
con.execute(f"""
COPY (
SELECT * FROM fake_data
WHERE category = '{category}'
)
TO '{file_path}'
(FORMAT PARQUET)
""")
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
