deniskuzZ opened a new issue, #16085:
URL: https://github.com/apache/iceberg/issues/16085
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Hive
### Please describe the bug 🐞
I encountered an issue where queries on Iceberg tables backed by ORC with
high number of position delete files (total 10mil rows, 2k delete files) hang
indefinitely when run multiple times.
Environment: Hive LLAP with 2 daemons, Iceberg ORC tables on S3 (AWS SDK
v2), Iceberg 1.5.2.
```
jmap -histo:live 1 | grep -i 'CPoolEntry\|S3AInput\|ManagedHttpClient'
51: 1513 205768
software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.LoggingManagedHttpClientConnection
79: 1513 145248
software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.CPoolEntry
430: 23 4784 org.apache.hadoop.fs.s3a.S3AInputStream
1420: 10 400
org.apache.hadoop.fs.s3a.S3AInputStream$$Lambda$4146/0x00007ee31ffa7cb0
2289: 3 144 org.apache.hadoop.fs.s3a.S3AInputPolicy
3110: 1 72
software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
3540: 1 48
[Lorg.apache.hadoop.fs.s3a.S3AInputPolicy;
```
A select count(*) on an Iceberg ORC table with delete files succeeds on the
first two executions (~10s each) but hangs forever on the third execution
within the same LLAP daemon.
It seems that when callers close the OrcIterable (via try-with-resources),
CloseableGroup.close() closes all registered closeables. The orcFileReader is
closed (releasing the metadata connection), but the rowBatchIterator is not.
Over multiple query executions, leaked connections accumulate and exhaust
the HTTP connection pool. When delete workers subsequently try to acquire
connections, the pool has none to give. The workers block indefinitely, and
the calling map tasks spin forever in Tasks.waitFor().
One liner change in OrcIterable resolved the issue;
VectorizedRowBatchIterator rowBatchIterator =
newOrcIterator(file, readOrcSchema, start, length, orcFileReader, sarg,
recordsPerBatch);
addCloseable(rowBatchIterator);
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
