adutra commented on code in PR #17914:
URL: https://github.com/apache/iceberg/pull/17914#discussion_r4584280181
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
@@ -380,11 +380,14 @@ interface Key {
String uri();
- static Key from(RemoteSignRequest request) {
+ Map<String, String> properties();
+
+ static Key from(RemoteSignRequest request, Map<String, String> properties)
{
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
+ .properties(properties)
Review Comment:
I would rather distinguish the request properties from the client
properties. Someone could in theory override `requestPropertiesSupplier`, even
if that's unlikely.
```suggestion
Map<String, String> requestProperties();
Map<String, String> clientProperties();
static Key from(RemoteSignRequest request, Map<String, String>
clientProperties) {
return ImmutableKey.builder()
.method(request.method())
.region(request.region())
.uri(request.uri().toString())
.requestProperties(request.properties())
.clientProperties(clientProperties)
```
##########
aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java:
##########
Review Comment:
I would add a javadoc here to explain the cache key logic, which imho is not
easy to grasp:
```suggestion
/**
* The key of the signed component cache. Besides the request itself, it
includes the properties
* of the signer client, since they determine how the request is signed:
signing endpoint,
* authentication, and remote signing configuration. The request
properties are included as well,
* since they do not necessarily derive from the client properties when
{@link
* S3V4RestSignerClient#requestPropertiesSupplier()} is overridden.
*/
@Value.Immutable
```
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri =
URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
+
+ ExecutionAttributes executionAttributes = new ExecutionAttributes();
+
executionAttributes.putAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME,
"s3");
Review Comment:
You may want to annotate this test with `@SuppressWarnings("deprecation")`
to suppress the warnings in lines 299-302.
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri =
URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
+
+ ExecutionAttributes executionAttributes = new ExecutionAttributes();
+
executionAttributes.putAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME,
"s3");
+
executionAttributes.putAttribute(AwsSignerExecutionAttribute.SIGNING_REGION,
Region.US_WEST_2);
+ executionAttributes.putAttribute(
+ AwsSignerExecutionAttribute.AWS_CREDENTIALS,
+ AnonymousCredentialsProvider.create().resolveCredentials());
+
+ AtomicInteger callCount = new AtomicInteger();
+ when(S3V4RestSignerClient.httpClient.post(
+ Mockito.eq("https://signer.com/v1/sign"),
+ Mockito.any(RemoteSignRequest.class),
+ Mockito.eq(RemoteSignResponse.class),
+ Mockito.anyMap(),
+ Mockito.any(),
+ Mockito.any()))
+ .thenAnswer(
+ invocation -> {
+ int count = callCount.incrementAndGet();
+ Consumer<Map<String, String>> headersConsumer =
invocation.getArgument(5);
+ if (headersConsumer != null) {
+ headersConsumer.accept(Map.of("Cache-Control", "private"));
+ }
+ return ImmutableRemoteSignResponse.builder()
+ .uri(requestUri)
+ .headers(Map.of("Authorization", List.of("AWS4-HMAC-SHA256
sig-client-" + count)))
+ .build();
+ });
+
+ try (S3V4RestSignerClient client1 =
+
ImmutableS3V4RestSignerClient.builder().properties(properties1).build();
+ S3V4RestSignerClient client2 =
+
ImmutableS3V4RestSignerClient.builder().properties(properties2).build()) {
+
+ // Client 1 makes request: cache miss, calls signer, response is cached
+ SdkHttpFullRequest signed1 = client1.sign(request, executionAttributes);
+ assertThat(signed1.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get()).isEqualTo(1);
+
+ // Client 1 repeated request: cache hit, does not call signer again
+ SdkHttpFullRequest signed1Repeat = client1.sign(request,
executionAttributes);
+ assertThat(signed1Repeat.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get()).isEqualTo(1);
+
+ // Client 2 makes the same request: must NOT reuse Client 1's cached
signature.
+ // It must call signer and receive Client 2's own signature.
+ SdkHttpFullRequest signed2 = client2.sign(request, executionAttributes);
+ assertThat(signed2.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-2");
+ assertThat(callCount.get()).isEqualTo(2);
+
+ // Client 2 repeated request: cache hit on client 2's cache
+ SdkHttpFullRequest signed2Repeat = client2.sign(request,
executionAttributes);
+ assertThat(signed2Repeat.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-2");
+ assertThat(callCount.get()).isEqualTo(2);
+
+ // Verify that another client instance with identical properties shares
the cache (e.g.
+ // multiple FileIOs for the same principal)
+ try (S3V4RestSignerClient client3 =
+
ImmutableS3V4RestSignerClient.builder().properties(properties1).build()) {
+ SdkHttpFullRequest signed3 = client3.sign(request,
executionAttributes);
+ assertThat(signed3.headers().get("Authorization"))
+ .containsExactly("AWS4-HMAC-SHA256 sig-client-1");
+ assertThat(callCount.get())
+ .as("Client 3 shares identical properties with Client 1 and must
hit the cache")
+ .isEqualTo(2);
+ }
+ }
+ }
Review Comment:
I would add two more tests to test that the endpoint and
`remote-signing-config` are also part of the cache key:
```java
@Test
void signedComponentCacheKeyIncludesRemoteSigningConfig() throws Exception
{
URI uri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/" +
UUID.randomUUID());
AtomicInteger signRequests = mockSigningServer(uri);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder().uri(uri).method(SdkHttpMethod.GET).build();
try (S3V4RestSignerClient client1 = signerClient("https://signer.com",
"value-1");
S3V4RestSignerClient client2 = signerClient("https://signer.com",
"value-2")) {
client1.sign(request, executionAttributes());
client2.sign(request, executionAttributes());
}
assertThat(signRequests.get())
.as("requests with different remote signing configs should be signed
separately")
.isEqualTo(2);
}
@Test
void signedComponentCacheKeyIncludesSigningEndpoint() throws Exception {
URI uri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/" +
UUID.randomUUID());
AtomicInteger signRequests = mockSigningServer(uri);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder().uri(uri).method(SdkHttpMethod.GET).build();
try (S3V4RestSignerClient client1 = signerClient("https://signer-1.com",
"value");
S3V4RestSignerClient client2 = signerClient("https://signer-2.com",
"value")) {
client1.sign(request, executionAttributes());
client2.sign(request, executionAttributes());
}
assertThat(signRequests.get())
.as("requests to different signing endpoints should be signed
separately")
.isEqualTo(2);
}
private static AtomicInteger mockSigningServer(URI uri) {
AtomicInteger signRequests = new AtomicInteger();
when(S3V4RestSignerClient.httpClient.post(
Mockito.anyString(),
Mockito.any(),
Mockito.eq(RemoteSignResponse.class),
Mockito.anyMap(),
Mockito.any(),
Mockito.any()))
.thenAnswer(
invocation -> {
signRequests.incrementAndGet();
Consumer<Map<String, String>> responseHeaders =
invocation.getArgument(5);
responseHeaders.accept(Map.of("Cache-Control", "private"));
return ImmutableRemoteSignResponse.builder()
.uri(uri)
.headers(Map.of("Authorization", List.of("AWS4-HMAC-SHA256
Credential=key")))
.build();
});
return signRequests;
}
@SuppressWarnings("deprecation")
private static ExecutionAttributes executionAttributes() {
return ExecutionAttributes.builder()
.put(
AwsSignerExecutionAttribute.AWS_CREDENTIALS,
AwsBasicCredentials.create("accessKeyId", "secretAccessKey"))
.put(AwsSignerExecutionAttribute.SIGNING_REGION, Region.US_WEST_2)
.put(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME, "s3")
.build();
}
private static S3V4RestSignerClient signerClient(String signerUri, String
propertyValue) {
return ImmutableS3V4RestSignerClient.builder()
.properties(
Map.of(
CatalogProperties.URI,
signerUri,
RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
"v1/namespaces/ns1/tables/t1/sign",
RESTCatalogProperties.REMOTE_SIGNING_CONFIG,
RemoteSigningConfigParser.toJson(
ImmutableRemoteSigningConfig.builder()
.putProperties("property", propertyValue)
.build()),
OAuth2Properties.TOKEN,
"token"))
.build();
}
```
##########
aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3V4RestSignerClient.java:
##########
@@ -262,4 +265,105 @@ public static Stream<Arguments> validOAuth2Properties() {
"custom",
"token"));
}
+
+ @Test
+ void doesNotReuseSignedComponentAcrossSignerClients() throws Exception {
+ Map<String, String> properties1 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token1");
+ Map<String, String> properties2 =
+ Map.of(
+ CatalogProperties.URI,
+ "https://signer.com",
+ RESTCatalogProperties.REMOTE_SIGNING_ENDPOINT,
+ "v1/sign",
+ OAuth2Properties.TOKEN,
+ "token2");
+
+ URI requestUri =
URI.create("https://bucket.s3.us-west-2.amazonaws.com/object-key");
+ SdkHttpFullRequest request =
+ SdkHttpFullRequest.builder()
+ .method(SdkHttpMethod.GET)
+ .uri(requestUri)
+ .protocol("https")
+ .host("bucket.s3.us-west-2.amazonaws.com")
+ .encodedPath("/object-key")
+ .build();
Review Comment:
Let's use a random object key for increased cross-test safety:
```suggestion
String objectKey = UUID.randomUUID().toString();
URI requestUri = URI.create("https://bucket.s3.us-west-2.amazonaws.com/"
+ objectKey);
SdkHttpFullRequest request =
SdkHttpFullRequest.builder()
.method(SdkHttpMethod.GET)
.uri(requestUri)
.protocol("https")
.host("bucket.s3.us-west-2.amazonaws.com")
.encodedPath("/" + objectKey)
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]