sunchao opened a new issue, #5502: URL: https://github.com/apache/datafusion-comet/issues/5502
### Describe the bug The process-wide object-store cache uses the normalized scheme/authority and configuration hash, but does not distinguish Hadoop from native backends. With `fs.comet.libhdfs.schemes=s3`, a Hadoop `s3://bucket/...` URL and a native `s3a://bucket/...` URL normalize to the same cache key under the same configuration. The second lookup can therefore reuse the first backend. The DataFusion registry has a related, separate limitation: its key contains only scheme/authority. Registering another configuration or backend at the same key replaces the previous mapping, even though the process cache has two distinct stores. ### Steps to reproduce Use one configuration with `fs.comet.libhdfs.schemes=s3`, then prepare both `s3://bucket/first.parquet` and `s3a://bucket/second.parquet`. Repeat in reverse order. The first cache insertion determines which backend the second URL receives. For the registry collision, prepare one bucket with two distinct object-store configurations in one `RuntimeEnv`, then retrieve both returned registration URLs. ### Expected behavior Cache and registration identities include the selected backend and configuration. Native `s3`/`s3a` aliases share one canonical identity, while explicitly Hadoop-routed schemes retain their physical spelling. Registration identity is independent of planning order. Synthetic registry identifiers must not reach physical encryption-key lookup. ### Additional context Extracted from the review of #5453 so this correctness change can be evaluated independently of I/O metric definitions and producer shutdown. The targeted regression uses distinct in-memory stores for deterministic cache/registry/read checks without requiring Hadoop or cloud credentials. This issue concerns object-store identity; file-metadata cache namespacing is not claimed to be addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
