dillitz commented on PR #18079: URL: https://github.com/apache/iceberg/pull/18079#issuecomment-6194343696
> 3. The URL replaces the location and flows through every path that treats `file_path` as a plain location, including the `files` metadata table and engine logs. I think that is acceptable for a first step; carrying the URL beside a native location would be a spec change to the task. I fear this extends a bit beyond the files table and logs. Besides being where we read a file from, the location is also often how other metadata refers to that file, and those references are compared as exact strings. So once the location becomes a URL, anything pointing at the file would need to point at that same URL too. For example, a DV says which data file it belongs to through `referenced-data-file`, and the reader checks that it matches the data file's location exactly ([here](https://github.com/apache/iceberg/blob/c3239b8622e6a4a941d4e6f1d2e5d5d67230b4c4/data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java#L279-L283)). So the server would also need to put the exact same signed string into every DV's `referenced-data-file`. Otherwise, the scan fails with `DV is expected to reference https://...` (+ the error message includes the signed URL). The DV test here handles this in the [test adapter](https://github.com/apache/iceberg/blob/ceafd63d95a60344f1da5f5e1f800ebc8fc3df78/core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java#L1791), but a real server wouldn't necessarily know it has to. DVs can probably be handled that way since `referenced-data-file` is part of the response, but I fear v2 position deletes can't: the data file paths are stored inside the delete file, so after reading it the client would compare native paths to the URL, and the deletes wouldn't apply. #17457 mentions positional deletes as unsupported, but as far as I can see nothing prevents a server from returning them. Because of that, I wonder whether carrying the URL alongside the native location would be worth doing from the start, so the URL is only used to fetch bytes and the native path remains for everything that identifies the file. #17332 had a similar idea (a path → URL map in the plan response, with `file-path` left untouched). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
