matanper commented on issue #68790: URL: https://github.com/apache/doris/issues/68790#issuecomment-6058340826
Correction to the last paragraph of the report: for **S3** vaults, `path_version=1` isn't supported past the FE either. I'd assumed the BE and meta-service already handled it; after tracing the code on 4.1.4, they don't. Fixing the FE `ImmutableMap.remove` lets the CREATE succeed, but the setting is then dropped silently: 1. **Meta-service:** the S3 vault branch persists only id, name and `obj_info`; `path_format` is not stored (`cloud/src/meta-service/meta_service_resource.cpp` ~1535–1541). The HDFS branch keeps it (~1555). 2. **BE:** `cloud_meta_mgr.cpp` (~1846–1848) builds S3 vaults with an empty path format, so they always use version 0. Only HDFS vaults use `vault.path_format()` (~1854). 3. **Recycler:** `cloud/src/recycler/util.h` (~64–92) hard-codes the version-0 layout `data/<tablet_id>/...`. If version-1 keys were ever written, the recycler would not delete them after DROP/TRUNCATE. 4. **File cache:** cache keys are derived from the file name only (`cached_remote_file_reader.cpp` ~136). Every version-1 segment is named `<seg>.dat`, so different tablets would collide in the cache unless the key includes the path. 5. `shard_num=0` with version 1 divides by zero in `storage_policy.cpp` (~166–169); nothing validates it. So the issue is really two parts: - (a) the FE crash, a one-line fix; - (b) S3 vaults have no working sharded-key layout. Making `path_version=1` usable for S3 needs coordinated changes in the meta-service (persist `path_format` for S3), the BE (use it), the recycler (version-1 key layout) and the file cache (path-qualified keys). Until (b) is done, it would be safer for the FE to reject `path_version`/`shard_num` for S3 vaults explicitly than to accept and drop them silently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
