milenkovicm commented on code in PR #1537:
URL:
https://github.com/apache/datafusion-ballista/pull/1537#discussion_r3143243044
##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -87,6 +90,8 @@ pub struct ShuffleWriterExec {
metrics: ExecutionPlanMetricsSet,
/// Plan properties
properties: Arc<PlanProperties>,
+ /// Bounded channel capacity for the async-to-blocking I/O bridge
Review Comment:
do we need channel capacity here if we can retrieve it from config?
##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -228,11 +243,11 @@ impl ShuffleWriterExec {
debug!("Writing results to {path:?}");
- // stream results to disk
let stats = utils::write_stream_to_disk(
&mut stream,
path.as_path(),
&write_metrics.write_time,
+ self.channel_capacity,
Review Comment:
i believe we could retrieve it from
```rust
let channel_capacity =
context.session_config().ballista_config().shuffle_writer_channel_capacity();
```
##########
ballista/executor/src/execution_engine.rs:
##########
@@ -106,6 +109,14 @@ impl ExecutionEngine for DefaultExecutionEngine {
})?
.data;
+ let channel_capacity = config
Review Comment:
i dont think we need this here, see previous comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
