raunaqmorarka commented on code in PR #18359:
URL: https://github.com/apache/iceberg/pull/18359#discussion_r4221060697
##########
core/src/main/java/org/apache/iceberg/DeleteFileIndex.java:
##########
@@ -519,6 +514,24 @@ private Collection<DeleteFile> loadDeleteFiles() {
return files;
}
+ /** Copies a delete file with the minimum stats needed for indexing to
limit memory use. */
+ private static DeleteFile copyWithMinStats(DeleteFile file) {
+ if (file.content() == FileContent.EQUALITY_DELETES) {
+ // copy with stats for better filtering against data file stats
+ return file.copyWithStats(Set.copyOf(file.equalityFieldIds()));
+ }
Review Comment:
These copies only reach callers through `FileScanTask.deletes()`. Metadata
tables open manifests directly and still see full stats. On main the index
already keeps only the `file_path` stats on these copies and drops the rest.
For a file-scoped delete, this PR moves that one value from the `file_path`
bounds into `referencedDataFile`. So `ContentFileUtil.referencedDataFile()` and
`isFileScoped()` return the same result. Position deletes that span several
data files keep their bounds.
##########
core/src/main/java/org/apache/iceberg/DeleteFileIndex.java:
##########
@@ -524,7 +527,8 @@ private static DeleteFile copyWithMinStats(DeleteFile file)
{
String referencedDataFile =
ContentFileUtil.referencedDataFileLocation(file);
if (referencedDataFile != null && file instanceof GenericDeleteFile) {
// a file-scoped position delete is matched by location and needs no
stats
- return ((GenericDeleteFile) file).copyWithoutStats(referencedDataFile);
+ String location =
referencedDataFiles.computeIfAbsent(referencedDataFile, key -> key);
Review Comment:
Yes. Each manifest entry decodes its own `String`, from either the
`file_path` bound or `referenced_data_file`. So N deletes of one data file hold
N equal copies. In the JOL run with 2.7 deletes per data file, sharing cuts
retained size from 717 B to 596 B per delete. The heap dump behind this PR had
about 11 deletes per referenced data file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
