marvinlanhenke commented on issue #124: URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2092098359
... so as a first step - simple wrap tokio::spawn (for example) like [here](https://github.com/launchbadge/sqlx/blob/main/sqlx-core/src/rt/mod.rs#L61-L78) - and not even use a feature-flag for now; just the most simplest layer of abstraction? @liurenjie1024 Have you already made up your mind; about how we want to read & filter the manifest files concurrently? The ideas I want to discuss would be (if they makes sense at all): Design 1: - iterate over manifest_list.entries() in an async stream on a single thread - apply manifest_evaluator - define `fn handle_data_file` where expression & inclusive_metrics evaluator are applied and return a stream of FileScanTask - spawn `fn handle_data_file` on multiple threads Design 2: - basically the same as 1 - but also spawn tasks for handling manifest_list.entries()? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
