morristai opened a new issue, #1890: URL: https://github.com/apache/iceberg-rust/issues/1890
### Apache Iceberg Rust version None ### Describe the bug ## Problem Description Follow up on https://github.com/apache/iceberg-rust/issues/1880, which I observed severe performance degradation and latency spikes during the `plan_files` stage. Root Cause: with `RUST_LOG=opendal=debug`, tracing reveals that the `OpenDAL/S3` backend operator is fully initialized for every concurrent manifest file read within the `fetch_manifest_and_stream_manifest_entries` function. This prevents the underlying HTTP client (`reqwest`/`hyper`) from reusing connections, forcing a new `TCP/TLS` handshake for every manifest file. I’m aware that there's an ongoing task to refactor iceberg-rust, focusing on the `FileIO` trait and the runtime API. But I’m just wondering if there’s any workaround for now—such as modifying underlaying implementations of `fetch_manifest_and_stream_manifest_entries` to use a shared S3 client? cc. @Xuanwo ## Debug Log ```log 2025-11-30T07:04:36.139691Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:36.139921Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:36.139933Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:36.141485Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:36.141689Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:36.308030Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ Step 1: Start getting manifest list... 2025-11-30T07:04:37.800722Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.800738Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.800741Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.801165Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.801179Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 Step 1 Done: Got manifest list in 134.06575ms Total manifests files: 24, largest manifest size: 8666105 (8.6MiB) Step 2 Done: Built 24 contexts in 448.333µs 2025-11-30T07:04:37.934764Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.934778Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.934781Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.935260Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.935274Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.935385Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.935392Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.935394Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.935744Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.935751Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.935802Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ 2025-11-30T07:04:37.935831Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.935835Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.935837Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.936285Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.936299Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.936359Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ 2025-11-30T07:04:37.936411Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.936419Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.936421Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.936728Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.936735Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.936823Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ 2025-11-30T07:04:37.936912Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:37.936916Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:37.936918Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.937237Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:37.937250Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:37.937302Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ 2025-11-30T07:04:38.171117Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.171133Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.171135Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.171478Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.171489Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.171544Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ 2025-11-30T07:04:38.391374Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.391389Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.391392Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.391810Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.391819Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.865136Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.865149Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.865151Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.865690Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.865700Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.865868Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.865873Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.865874Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.866237Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.866242Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.866305Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.866310Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.866311Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.866731Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.866735Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.866839Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.866843Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.866845Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.867197Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.867201Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.867300Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:38.867303Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:38.867305Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:38.867631Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:38.867635Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.747231Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:39.747254Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:39.747256Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.747655Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:39.747666Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.747936Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:39.747940Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:39.747941Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.748233Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:39.748238Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.748556Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:39.748560Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:39.748561Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.749024Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:39.749029Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.749079Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:39.749083Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:39.749084Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.749427Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:39.749431Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.749509Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:39.749513Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:39.749514Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:39.749839Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:39.749843Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.005749Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:40.005765Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:40.005767Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.006203Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:40.006214Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.065351Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:40.065365Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:40.065367Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.065676Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:40.065687Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.767797Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:40.767813Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:40.767815Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.768182Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:40.768191Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.768210Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:40.768213Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:40.768214Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.768406Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:40.768411Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.769792Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:40.769799Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:40.769801Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:40.770026Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:40.770034Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 --> Slow manifest fetch: 16.169187291s 2025-11-30T07:04:56.175095Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:56.175105Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:56.175107Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:56.176108Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:56.176124Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 --> Slow manifest fetch: 15.40907375s 2025-11-30T07:04:56.178985Z DEBUG opendal::services::s3::backend: backend build started: S3Builder { config: S3Config { root: None, bucket: "crem-lakehouse-398360765845-eu-central-1", endpoint: None, region: Some("eu-central-1"), .. }, .. } 2025-11-30T07:04:56.178989Z DEBUG opendal::services::s3::backend: backend use root / 2025-11-30T07:04:56.178990Z DEBUG opendal::services::s3::backend: backend use bucket crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:56.179332Z DEBUG opendal::services::s3::backend: backend use region: eu-central-1 2025-11-30T07:04:56.179341Z DEBUG opendal::services::s3::backend: backend use endpoint: https://s3.eu-central-1.amazonaws.com/crem-lakehouse-398360765845-eu-central-1 2025-11-30T07:04:56.179396Z DEBUG reqwest::connect: starting new connection: https://s3.eu-central-1.amazonaws.com/ --> Slow manifest fetch: 16.121498333s --> Slow manifest fetch: 15.423606375s --> Slow manifest fetch: 15.426948791s --> Slow manifest fetch: 4.16883575s Step 3 Done: All manifests fetched in 22.409096083s Planned 151 file scan tasks in 22.547563667s ``` ### To Reproduce refer to https://github.com/apache/iceberg-rust/issues/1880. ### Expected behavior _No response_ ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
