stevenzwu commented on code in PR #10859:
URL: https://github.com/apache/iceberg/pull/10859#discussion_r1706409470
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -233,15 +239,56 @@ public Builder flinkConf(ReadableConfig config) {
* @return {@link Builder} to connect the iceberg table.
*/
public Builder distributionMode(DistributionMode mode) {
- Preconditions.checkArgument(
- !DistributionMode.RANGE.equals(mode),
- "Flink does not support 'range' write distribution mode now.");
if (mode != null) {
writeOptions.put(FlinkWriteOptions.DISTRIBUTION_MODE.key(),
mode.modeName());
}
return this;
}
+ /**
+ * Range distribution needs to collect statistics about data distribution
to properly shuffle
+ * the records in relatively balanced way. In general, low cardinality
should use {@link
+ * StatisticsType#Map} and high cardinality should use {@link
StatisticsType#Sketch} Refer to
+ * {@link StatisticsType} Javadoc for more details.
+ *
+ * <p>Default is {@link StatisticsType#Auto} where initially Map
statistics is used. But if
+ * cardinality is higher than some threshold (like 10K), statistics
collection automatically
Review Comment:
hmm. thought it is a good idea. but just realized that the class is package
private. We would need to make the class and the constant public
`SketchUtil#COORDINATOR_SKETCH_SWITCH_THRESHOLD`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
