aokolnychyi commented on code in PR #8972:
URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380713525
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java:
##########
@@ -221,41 +206,24 @@ private Dataset<Row>
buildManifestEntryDF(List<ManifestFile> manifests) {
private List<ManifestFile> writeManifestsForUnpartitionedTable(
Dataset<Row> manifestEntryDF, int numManifests) {
- Broadcast<Table> tableBroadcast =
- sparkContext().broadcast(SerializableTableWithSize.copyOf(table));
StructType sparkType = (StructType)
manifestEntryDF.schema().apply("data_file").dataType();
Types.StructType combinedPartitionType = Partitioning.partitionType(table);
-
- // we rely only on the target number of manifests for unpartitioned tables
- // as we should not worry about having too much metadata per partition
- long maxNumManifestEntries = Long.MAX_VALUE;
+ Types.StructType partitionType = spec.partitionType();
return manifestEntryDF
.repartition(numManifests)
Review Comment:
I think using the advisory partition size is going to be tricky given that
it is expressed in terms of shuffle files. I feel our own estimation would
still be much more precise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
