github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4357160361
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3SourceOffsetProvider.java:
##########
@@ -91,7 +91,8 @@ public S3Offset getNextOffset(StreamingJobProperties
jobProps, Map<String, Strin
String beginFile =
rfiles.get(0).location().uri().replace(bucketBase, "");
String lastFile = rfiles.get(rfiles.size() -
1).location().uri().replace(bucketBase, "");
offset.setFileStatuses(rfiles.stream().map(entry -> new
TBrokerFileStatus(
- entry.location().uri(), entry.isDirectory(),
entry.length(), !entry.isDirectory()))
+ entry.location().uri(), entry.isDirectory(),
entry.length(), !entry.isDirectory())
+ .setModificationTime(entry.modificationTime()))
Review Comment:
[P2] Scope native cache identities to the configured S3 endpoint
The newly nonzero mtime enables V2 Parquet's native footer/page caches for
streaming LIST files, but their key uses only fs_name, path, mtime and size.
For FILE_S3, fs_name is empty and S3FileReader's path is just s3://bucket/key;
FileFactory adds the configured endpoint only to the lower block-cache key. On
the same BE, two S3-compatible endpoints containing different objects at the
same bucket/key with equal listed mtime/size therefore share native metadata.
If A warms a footer with id min/max 1 and B contains id=2 with the same schema,
B's WHERE id=2 can prune its matching row group even while both objects remain
unchanged. Before this assignment, streaming's zero mtime bypassed these
mutable-file native caches. This differs from the existing same-source
overwrite collision: include the endpoint/storage namespace in the native
identity, or keep these caches bypassed until it is available, and cover two
endpoints with matching object metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
