pawel-big-lebowski opened a new issue, #11664:
URL: https://github.com/apache/iceberg/issues/11664
### Apache Iceberg version
1.6.0
### Query engine
Spark
### Please describe the bug 🐞
I am running Spark jobs locally on my laptop with a custom MetricsReporter
which does no external connections and is pretty much same to in-memory metrics
reporter.
My SQL is as simple as:
```
spark.sql("CREATE TABLE source1 USING iceberg AS SELECT * FROM temp");
spark.sql("CREATE TABLE target USING iceberg AS SELECT * FROM source1");
```
with a table having two fields and two rows.
In this case, I see my custom metrics reporter got loaded due to the log:
```
2024-11-26 15:11:10 INFO org.apache.iceberg.CatalogUtil - Loading custom
MetricsReporter implementation:
io.openlineage.spark.agent.vendor.iceberg.metrics.OpenLineageMetricsReporter
```
I am able to debug that my metrics reporter gets `ScanReport`. However, it
is not getting any `CommitReport`. Instead, they're logged with
`LoggingMetricsReporter`:
```
2024-11-26 15:11:12 INFO org.apache.iceberg.metrics.LoggingMetricsReporter
- Received metrics report: CommitReport{tableName=default.source1,
snapshotId=2507000462464497081, sequenceNumber=1, ....,
metadata={engine-version=3.3.4, app-id=local-1732630268734, engine-name=spark,
iceberg-version=Apache Iceberg 1.6.0 (commit
229d8f6fcd109e6c8943ea7cbb41dab746c6d0ed)}}
```
There are no logs of the form `Cannot initialize MetricsReporter`.
Moreover, when I run the query:
```
spark.sql("INSERT INTO target VALUES (4, 5)");
```
I see commit reports within my custom reporter. That's why I think it's a
bug while not being a configuration issue on my side.
During short debugging, I found out that `BaseTransaction` object is
initialised through a constructor
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L82
which injects `LoggingMetricsReporter` instead of using the one configured.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
