abjain123 opened a new issue, #18292:
URL: https://github.com/apache/iceberg/issues/18292
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When querying Iceberg V3 tables in Spark, row-lineage metadata columns
cannot be used in SQL aggregates when aggregate push-down is enabled. The
queries fail at query-planning time with a ValidationException. The same
queries succeed when aggregate push-down is disabled via
spark.sql.iceberg.aggregate-push-down.enabled=false.
**_Steps to Reproduce:_**
1. Initialize a PySpark session with the Iceberg catalog and extensions
enabled.
2. Create an Iceberg table with 'format-version'='3'.
3. Insert mock data.
4. Execute an aggregate function on _row_id
```
CAT = "ds2_standalone"
DB = "iceberg_v3_qe"
NS = f"{CAT}.{DB}"
TBL_SHORT = "v3_tc014_v3"
TBL = f"{NS}.{TBL_SHORT}"
OPTS = "USING ICEBERG TBLPROPERTIES ('format-version'='3',
'write.format.default'='parquet')"
spark.sql(f"CREATE DATABASE IF NOT EXISTS {NS}")
spark.sql(f"DROP TABLE IF EXISTS {TBL}")
spark.sql(f"CREATE TABLE {TBL} (id INT, name STRING) {OPTS}")
spark.sql(f"INSERT INTO {TBL} VALUES (1, 'Alice'), (2, 'Bob'), (3, 'Carol')")
spark.sql(f"SELECT _row_id from {TBL}").show() # Works
spark.sql(f"SELECT MAX(_row_id) FROM {TBL}").show() # Fails here
```
**_Metrics:_**
**Spark Version: 4.1.1**
**Iceberg Table Format: V3**
**_Error Trace:_**
`org.apache.iceberg.exceptions.ValidationException: Cannot find field
'_row_id' in struct: struct<1: id: optional int, 2: name: optional string>`
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
