alexanderbianchi commented on code in PR #3514:
URL: https://github.com/apache/iceberg-rust/pull/3514#discussion_r4407283091
##########
crates/iceberg/src/arrow/reader/mod.rs:
##########
@@ -133,33 +127,53 @@ impl ArrowReaderBuilder {
///
/// Defaults to 1 MiB, matching object_store's
OBJECT_STORE_COALESCE_DEFAULT.
pub fn with_range_coalesce_bytes(mut self, range_coalesce_bytes: u64) ->
Self {
- self.parquet_read_options.range_coalesce_bytes = range_coalesce_bytes;
+ self.options
+ .parquet_options_mut()
+ .set_range_coalesce_bytes(range_coalesce_bytes);
self
}
/// Sets the maximum number of merged byte ranges to fetch concurrently.
///
/// Defaults to 10, matching object_store's OBJECT_STORE_COALESCE_PARALLEL.
pub fn with_range_fetch_concurrency(mut self, range_fetch_concurrency:
usize) -> Self {
- self.parquet_read_options.range_fetch_concurrency =
range_fetch_concurrency;
+ self.options
+ .parquet_options_mut()
+ .set_range_fetch_concurrency(range_fetch_concurrency);
+ self
+ }
+
+ /// Replaces all reader settings while preserving local I/O and runtime
resources.
+ pub fn with_options(mut self, options: ArrowReaderOptions) -> Self {
+ self.options = options;
+ self
+ }
+
+ /// Replaces Parquet I/O settings while preserving other reader settings.
+ pub fn with_parquet_options(mut self, options: ParquetReadOptions) -> Self
{
+ self.options.set_parquet_options(options);
self
}
/// Build the ArrowReader.
pub fn build(self) -> ArrowReader {
+ let concurrency_limit_data_files = self
+ .options
+ .data_file_concurrency_limit()
+ .unwrap_or_else(|| available_parallelism().get());
Review Comment:
IMO `.max(1)` makes sense here.
Zero becomes sequential execution instead of hanging. It keeps construction
infallible, handles deserialized values, and matches the existing treatment of
zero Parquet range-fetch concurrency.
NonZeroUsize is stricter, but would require callers to construct that type
and make zero fail deserialization. That’s reasonable if we want invalid
configuration rejected; for this PR, normalization is the smaller change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
