raviston commented on PR #18265: URL: https://github.com/apache/iceberg/pull/18265#issuecomment-6948821926
This was closed by the stale bot, but #18263 still reproduces on `main`, and this fix still applies cleanly. @ragnard, could you reopen it? I checked it with a unit test that has one mock `S3Client` per prefix: credentials for `s3://bucket/table1` and `s3://bucket/table2`, and `s3.delete.batch-size=5`. Then I compared which client each key was sent to. | Files deleted | `main` | With this PR | |---|---|---| | 7 in table1 | the table1 client gets 5 keys; the **root client** (no vended credentials) gets the last 2 | the table1 client gets all 7 | | 7 in table1 + 3 in table2 | the table1 client gets 5 keys; the **table2 client** gets table1/f5, table1/f6 and the 3 table2 keys | each client gets only its own table's keys | So there's a second case besides the remainder going to the root client. On `main`, batches are grouped only by bucket, so one batch can hold keys from several prefixes. It is then sent with the client of whichever path filled it. So any `deleteFiles` call whose paths span more than one credential prefix can send some keys with another prefix's credentials. Grouping by client as well, as this PR does, fixes both cases. The rest of `:iceberg-aws:test` passes with the PR (241 tests). I couldn't run the new `TestS3FileIO` integration test locally (no Docker). @uros-b approved this. cc @nastra, could you take a look when you have a chance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
