phillipleblanc commented on code in PR #2039:
URL:
https://github.com/apache/datafusion-ballista/pull/2039#discussion_r3583880534
##########
ballista/core/src/config.rs:
##########
@@ -334,6 +290,14 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
DataType::Utf8,
Some("".to_string()),
),
+ ConfigEntry::new(
+ BALLISTA_SHUFFLE_COMPRESSION_CODEC.to_string(),
+ "Compression codec specification \
+ used in the shuffle process. Possible values: \
+ none, lz4, zstd. Default to lz4 to preserve current
behaviour".to_string(),
Review Comment:
```suggestion
none, lz4, zstd. Defaults to lz4 to preserve current
behaviour".to_string(),
```
nit: grammar
##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -315,10 +311,10 @@ impl ShuffleWriterExec {
debug!("Writing results to {p:?}");
- let options =
IpcWriteOptions::default()
- .try_with_compression(Some(
- CompressionType::LZ4_FRAME,
- ))?;
+ let options =
+
create_new_write_options(config.clone())
Review Comment:
This should propagate the error instead of doing unwrap
##########
ballista/core/src/config.rs:
##########
@@ -585,6 +517,11 @@ impl BallistaConfig {
self.get_bool_setting(BALLISTA_PROPAGATE_EMPTY_ENABLED)
}
+ /// Returns compression codec that will be used during write stage of
shuffle
+ pub fn shuffle_compression_codec(&self) -> String {
+ self.get_string_setting(BALLISTA_SHUFFLE_COMPRESSION_CODEC)
Review Comment:
Maybe we could return Result<CompressionType> here and do the parsing here
instead of in `create_new_write_options`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
