This is an automated email from the ASF dual-hosted git repository.
cryptoe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 0849af5e5df minor: Add new intermediate S3 role (#20540)
0849af5e5df is described below
commit 0849af5e5df35e4bb602d28f84429c243989687f
Author: Adarsh Sanjeev <[email protected]>
AuthorDate: Fri Oct 23 23:29:54 2026 +0530
minor: Add new intermediate S3 role (#20540)
* Add new intermediate chained role
* Fix test
* Add test
* Builds
* Address comments
* Address comments
* Update doc
---
docs/development/extensions-core/s3.md | 19 +++
docs/ingestion/input-sources.md | 2 +-
docs/multi-stage-query/reference.md | 2 +-
extensions-core/s3-extensions/pom.xml | 5 +
.../apache/druid/data/input/s3/S3InputSource.java | 4 +-
.../druid/storage/s3/S3IntermediateRoleConfig.java | 59 ++++++++
.../druid/storage/s3/S3StorageDruidModule.java | 8 +-
.../storage/s3/ServerSideEncryptingAmazonS3.java | 70 ++++++++-
.../storage/s3/output/S3ExportStorageProvider.java | 8 +-
.../druid/data/input/s3/S3InputSourceTest.java | 10 ++
.../storage/s3/S3IntermediateRoleConfigTest.java | 104 ++++++++++++++
...rSideEncryptingAmazonS3MultipartUploadTest.java | 2 +
.../s3/ServerSideEncryptingAmazonS3Test.java | 157 +++++++++++++++++++++
.../storage/s3/TestAWSCredentialsProvider.java | 6 +-
14 files changed, 446 insertions(+), 10 deletions(-)
diff --git a/docs/development/extensions-core/s3.md
b/docs/development/extensions-core/s3.md
index 43b225aabff..6fc754c267c 100644
--- a/docs/development/extensions-core/s3.md
+++ b/docs/development/extensions-core/s3.md
@@ -81,6 +81,24 @@ For more information, refer to the [Amazon Developer
Guide](https://docs.aws.ama
Alternatively, you can bypass this chain by specifying an access key and
secret key using a [Properties
Object](../../ingestion/input-sources.md#s3-input-source) inside your ingestion
specification.
+Whichever credentials are resolved are then used to assume any `assumeRoleArn`
named by an ingestion spec or export destination. Setting
`druid.s3.intermediateAssumeRoleArn` inserts one role in between: the resolved
credentials assume that role first, and the resulting credentials assume
`assumeRoleArn`. Keys given in an ingestion spec's Properties Object skip the
intermediate role and assume `assumeRoleArn` directly.
+
+Set `druid.s3.intermediateAssumeRoleArn` on every process that builds an S3
input source or export destination:
+
+* Peons, which run ingestion and export tasks. Peons inherit the Middle
Manager's runtime properties, so set it in the Middle Manager's
`runtime.properties`, or set
`druid.indexer.fork.property.druid.s3.intermediateAssumeRoleArn` there to apply
it to Peons only.
+* Indexers, which run tasks in their own process.
+* The Overlord, which runs the sampler used by the web console's data loader.
+
+A process that doesn't have the property set keeps assuming `assumeRoleArn`
directly, so leaving it out of one of these causes the same ingestion to
succeed on some processes and fail on others.
+
+:::warning
+
+Scope an intermediate role to a single trust boundary. Druid assumes it with
no external ID, so every role that trusts it is reachable by every ingestion
spec and `EXPORT` statement that any process using the role can run. If several
clusters share one ARN, the bucket owner's trust policy can no longer tell
those clusters apart, and a user of any one of them can assume any role that
trusts the shared intermediate role.
+
+Note that `assumeRoleArn` and `assumeRoleExternalId` both come from whoever
writes the spec or `EXPORT` statement, so the external ID doesn't restrict
which roles a spec author can reach. Treat it as a value spec authors supply
rather than a secret the cluster holds, and have the bucket owner's trust
policy turn on something spec authors don't control, such as an intermediate
role used by only one cluster.
+
+:::
+
Use the property
[`druid.startup.logging.maskProperties`](../../configuration/index.md#startup-logging)
to mask credentials information in Druid logs. For example, `["password",
"secretKey", "awsSecretAccessKey"]`.
### S3 permissions settings
@@ -127,6 +145,7 @@ For example, to set the region to 'us-east-1' through
system properties:
|`druid.s3.accessKey`|S3 access key. See [S3 authentication
methods](#s3-authentication-methods) for more details|Can be omitted according
to authentication methods chosen.|
|`druid.s3.secretKey`|S3 secret key. See [S3 authentication
methods](#s3-authentication-methods) for more details|Can be omitted according
to authentication methods chosen.|
|`druid.s3.fileSessionCredentials`|Path to properties file containing
`sessionToken`, `accessKey` and `secretKey` value. One key/value pair per line
(format `key=value`). See [S3 authentication
methods](#s3-authentication-methods) for more details |Can be omitted according
to authentication methods chosen.|
+|`druid.s3.intermediateAssumeRoleArn`|AWS ARN of a role to assume before
assuming the `assumeRoleArn` given by an ingestion spec or export destination.
Lets a bucket owner's trust policy name one fixed ARN instead of a different
principal for every Druid cluster. See [S3 authentication
methods](#s3-authentication-methods) for more details.|None|
|`druid.s3.protocol`|Communication protocol type to use when sending requests
to AWS. `http` or `https` can be used. This configuration would be ignored if
`druid.s3.endpoint.url` is filled with a URL with a different protocol.|`https`|
|`druid.s3.disableChunkedEncoding`|Disables chunked encoding. See [AWS
document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#disableChunkedEncoding--)
for details.|false|
|`druid.s3.enablePathStyleAccess`|Enables path style access. See [AWS
document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#enablePathStyleAccess--)
for details.|false|
diff --git a/docs/ingestion/input-sources.md b/docs/ingestion/input-sources.md
index 05b019ef3f3..99646c5503c 100644
--- a/docs/ingestion/input-sources.md
+++ b/docs/ingestion/input-sources.md
@@ -212,7 +212,7 @@ Properties Object:
:::info
-If `accessKeyId` and `secretAccessKey` are not given, the default [S3
credentials provider
chain](../development/extensions-core/s3.md#s3-authentication-methods) is used.
+If `accessKeyId` and `secretAccessKey` are not given, the default [S3
credentials provider
chain](../development/extensions-core/s3.md#s3-authentication-methods) is used.
If the Druid process sets `druid.s3.intermediateAssumeRoleArn`, the resolved
credentials assume that role before assuming `assumeRoleArn`.
:::
diff --git a/docs/multi-stage-query/reference.md
b/docs/multi-stage-query/reference.md
index a774dc11fdf..739a4ea3599 100644
--- a/docs/multi-stage-query/reference.md
+++ b/docs/multi-stage-query/reference.md
@@ -152,7 +152,7 @@ Supported arguments for the function:
|---|---|---|---|
| `bucket` | Yes | S3 bucket destination for exported files. You must add the
bucket and prefix combination to the
`druid.export.storage.s3.allowedExportPaths` allow list. | n/a |
| `prefix` | Yes | Destination path in the bucket to create exported files.
The export query expects the destination path to be empty. If the location
includes other files, the query will fail. You must add the bucket and prefix
combination to the `druid.export.storage.s3.allowedExportPaths` allow list. |
n/a |
-| `assumeRoleArn` | No | ARN of the role to assume before exporting data. If
not provided, the default credentials configured for the Druid process are
used. | n/a |
+| `assumeRoleArn` | No | ARN of the role to assume before exporting data. If
not provided, the default credentials configured for the Druid process are
used. If provided and the Druid process sets
`druid.s3.intermediateAssumeRoleArn`, those credentials assume that role before
assuming `assumeRoleArn`. | n/a |
| `assumeRoleExternalId` | No | External ID to use when assuming the role
specified in assumeRoleArn. This provides an additional layer of security for
role assumption. Only used when assumeRoleArn is set. | n/a |
Configure the following runtime parameters to export to an S3 destination:
diff --git a/extensions-core/s3-extensions/pom.xml
b/extensions-core/s3-extensions/pom.xml
index 584d6c663dc..976ce809ea3 100644
--- a/extensions-core/s3-extensions/pom.xml
+++ b/extensions-core/s3-extensions/pom.xml
@@ -142,6 +142,11 @@
<artifactId>auth</artifactId>
<version>${aws.sdk.v2.version}</version>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>identity-spi</artifactId>
+ <version>${aws.sdk.v2.version}</version>
+ </dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sdk-core</artifactId>
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSource.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSource.java
index 85cd832d98f..9164c9bad29 100644
---
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSource.java
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSource.java
@@ -129,7 +129,9 @@ public class S3InputSource extends CloudObjectInputSource
awsEndpointConfig,
awsClientConfig,
s3InputSourceConfig,
- null
+ null,
+ s3ClientBuilder.getIntermediateRoleConfig(),
+ s3ClientBuilder.getIntermediateRoleEndpointConfig()
).build();
});
this.maxRetries = RetryUtils.DEFAULT_MAX_TRIES;
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3IntermediateRoleConfig.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3IntermediateRoleConfig.java
new file mode 100644
index 00000000000..9124eef1d46
--- /dev/null
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3IntermediateRoleConfig.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.annotations.VisibleForTesting;
+
+import javax.annotation.Nullable;
+
+/**
+ * Role assumed before the {@code assumeRoleArn} named by an ingestion spec or
export destination
+ */
+public class S3IntermediateRoleConfig
+{
+ @Nullable
+ @JsonProperty
+ private String intermediateAssumeRoleArn;
+
+ public S3IntermediateRoleConfig()
+ {
+ }
+
+ @VisibleForTesting
+ S3IntermediateRoleConfig(@Nullable String intermediateAssumeRoleArn)
+ {
+ this.intermediateAssumeRoleArn = intermediateAssumeRoleArn;
+ }
+
+ @Nullable
+ public String getIntermediateAssumeRoleArn()
+ {
+ return intermediateAssumeRoleArn;
+ }
+
+ @Override
+ public String toString()
+ {
+ return "S3IntermediateRoleConfig{" +
+ "intermediateAssumeRoleArn='" + intermediateAssumeRoleArn + '\'' +
+ '}';
+ }
+}
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java
index a0c7f986655..8ada5b50165 100644
---
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java
@@ -116,6 +116,7 @@ public class S3StorageDruidModule implements DruidModule
JsonConfigProvider.bind(binder, "druid.storage", S3StorageConfig.class);
JsonConfigProvider.bind(binder, "druid.storage.sse.kms",
S3SSEKmsConfig.class);
JsonConfigProvider.bind(binder, "druid.storage.sse.custom",
S3SSECustomConfig.class);
+ JsonConfigProvider.bind(binder, "druid.s3",
S3IntermediateRoleConfig.class);
Binders.bindTaskLogs(binder, SCHEME, S3TaskLogs.class);
JsonConfigProvider.bind(binder, "druid.indexer.logs",
S3TaskLogsConfig.class);
@@ -130,7 +131,8 @@ public class S3StorageDruidModule implements DruidModule
AWSProxyConfig proxyConfig,
AWSEndpointConfig endpointConfig,
AWSClientConfig clientConfig,
- S3StorageConfig storageConfig
+ S3StorageConfig storageConfig,
+ S3IntermediateRoleConfig intermediateRoleConfig
)
{
if (clientConfig.isForceGlobalBucketAccessEnabled() != null) {
@@ -149,7 +151,9 @@ public class S3StorageDruidModule implements DruidModule
endpointConfig,
clientConfig,
null,
- null
+ null,
+ intermediateRoleConfig,
+ endpointConfig
);
}
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java
index 81884ea0cf4..133ab8f8d89 100644
---
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java
@@ -322,7 +322,9 @@ public class ServerSideEncryptingAmazonS3
@Nullable AWSEndpointConfig awsEndpointConfig,
@Nullable AWSClientConfig awsClientConfig,
@Nullable S3InputSourceConfig s3InputSourceConfig,
- @Nullable S3ExportStorageProvider s3ExportStorageProvider
+ @Nullable S3ExportStorageProvider s3ExportStorageProvider,
+ @Nullable S3IntermediateRoleConfig intermediateRoleConfig,
+ @Nullable AWSEndpointConfig intermediateRoleEndpointConfig
)
{
if (s3InputSourceConfig != null && s3ExportStorageProvider != null) {
@@ -411,6 +413,15 @@ public class ServerSideEncryptingAmazonS3
// Apply assume role if configured
if (!Strings.isNullOrEmpty(assumeRoleArn)) {
+ if (s3InputSourceConfig == null ||
!s3InputSourceConfig.isCredentialsConfigured()) {
+ credentialsProvider = maybeAssumeIntermediateRole(
+ credentialsProvider,
+ intermediateRoleConfig,
+ intermediateRoleEndpointConfig,
+ awsClientConfig
+ );
+ }
+
credentialsProvider = createAssumeRoleCredentialsProvider(
assumeRoleArn,
assumeRoleExternalId,
@@ -427,7 +438,9 @@ public class ServerSideEncryptingAmazonS3
return ServerSideEncryptingAmazonS3.builder()
.setS3ClientSupplier(clientBuilder::build)
.setS3AsyncClientSupplier(asyncClientBuilder::build)
- .setS3StorageConfig(s3StorageConfig);
+ .setS3StorageConfig(s3StorageConfig)
+
.setIntermediateRoleConfig(intermediateRoleConfig)
+
.setIntermediateRoleEndpointConfig(intermediateRoleEndpointConfig);
}
/**
@@ -461,6 +474,31 @@ public class ServerSideEncryptingAmazonS3
}
}
+ /**
+ * Assumes an intermediate role if one is configured.
+ */
+ private static AwsCredentialsProvider maybeAssumeIntermediateRole(
+ AwsCredentialsProvider baseCredentialsProvider,
+ @Nullable S3IntermediateRoleConfig intermediateRoleConfig,
+ @Nullable AWSEndpointConfig intermediateRoleEndpointConfig,
+ @Nullable AWSClientConfig awsClientConfig
+ )
+ {
+ final String intermediateAssumeRoleArn =
+ intermediateRoleConfig == null ? null :
intermediateRoleConfig.getIntermediateAssumeRoleArn();
+ if (Strings.isNullOrEmpty(intermediateAssumeRoleArn)) {
+ return baseCredentialsProvider;
+ }
+
+ return createAssumeRoleCredentialsProvider(
+ intermediateAssumeRoleArn,
+ null,
+ intermediateRoleEndpointConfig,
+ baseCredentialsProvider,
+ awsClientConfig
+ );
+ }
+
public static AwsCredentialsProvider createAssumeRoleCredentialsProvider(
String assumeRoleArn,
@Nullable String assumeRoleExternalId,
@@ -500,6 +538,10 @@ public class ServerSideEncryptingAmazonS3
@Nullable
private Supplier<S3AsyncClient> s3AsyncClientSupplier;
private S3StorageConfig s3StorageConfig = new S3StorageConfig(new
NoopServerSideEncryption(), null);
+ @Nullable
+ private S3IntermediateRoleConfig intermediateRoleConfig;
+ @Nullable
+ private AWSEndpointConfig intermediateRoleEndpointConfig;
public Builder setS3ClientSupplier(Supplier<S3Client> s3ClientSupplier)
{
@@ -524,6 +566,30 @@ public class ServerSideEncryptingAmazonS3
return this.s3StorageConfig;
}
+ public Builder setIntermediateRoleConfig(@Nullable
S3IntermediateRoleConfig intermediateRoleConfig)
+ {
+ this.intermediateRoleConfig = intermediateRoleConfig;
+ return this;
+ }
+
+ @Nullable
+ public S3IntermediateRoleConfig getIntermediateRoleConfig()
+ {
+ return this.intermediateRoleConfig;
+ }
+
+ public Builder setIntermediateRoleEndpointConfig(@Nullable
AWSEndpointConfig intermediateRoleEndpointConfig)
+ {
+ this.intermediateRoleEndpointConfig = intermediateRoleEndpointConfig;
+ return this;
+ }
+
+ @Nullable
+ public AWSEndpointConfig getIntermediateRoleEndpointConfig()
+ {
+ return this.intermediateRoleEndpointConfig;
+ }
+
/**
* Builds a new {@link ServerSideEncryptingAmazonS3} instance.
*
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3ExportStorageProvider.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3ExportStorageProvider.java
index cdcd79b1675..566d06c869c 100644
---
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3ExportStorageProvider.java
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3ExportStorageProvider.java
@@ -40,6 +40,7 @@ import org.apache.druid.error.DruidException;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.storage.ExportStorageProvider;
import org.apache.druid.storage.StorageConnector;
+import org.apache.druid.storage.s3.S3IntermediateRoleConfig;
import org.apache.druid.storage.s3.S3StorageConfig;
import org.apache.druid.storage.s3.S3StorageDruidModule;
import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
@@ -85,6 +86,9 @@ public class S3ExportStorageProvider implements
ExportStorageProvider
AWSClientConfig awsClientConfig;
@JacksonInject
AwsCredentialsProvider baseCredentialsProvider;
+ @JacksonInject
+ @Nullable
+ S3IntermediateRoleConfig intermediateRoleConfig;
@JacksonInject
S3UploadManager s3UploadManager;
@@ -113,7 +117,9 @@ public class S3ExportStorageProvider implements
ExportStorageProvider
awsEndpointConfig,
awsClientConfig,
null,
- this
+ this,
+ intermediateRoleConfig,
+ awsEndpointConfig
).build();
});
}
diff --git
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/data/input/s3/S3InputSourceTest.java
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/data/input/s3/S3InputSourceTest.java
index 0e041a7640d..421f7eb7fb3 100644
---
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/data/input/s3/S3InputSourceTest.java
+++
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/data/input/s3/S3InputSourceTest.java
@@ -348,6 +348,8 @@ public class S3InputSourceTest extends
InitializedNullHandlingTest
{
EasyMock.reset(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getS3StorageConfig()).andStubReturn(S3_STORAGE_CONFIG);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleConfig()).andStubReturn(null);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleEndpointConfig()).andStubReturn(null);
EasyMock.replay(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
final S3InputSource withPrefixes = new S3InputSource(
SERVICE,
@@ -375,6 +377,8 @@ public class S3InputSourceTest extends
InitializedNullHandlingTest
{
EasyMock.reset(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getS3StorageConfig()).andStubReturn(S3_STORAGE_CONFIG);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleConfig()).andStubReturn(null);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleEndpointConfig()).andStubReturn(null);
EasyMock.replay(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
final S3InputSource withSessionToken = new S3InputSource(
SERVICE,
@@ -407,6 +411,8 @@ public class S3InputSourceTest extends
InitializedNullHandlingTest
EasyMock.reset(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getS3StorageConfig())
.andStubReturn(S3_STORAGE_CONFIG);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleConfig()).andStubReturn(null);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleEndpointConfig()).andStubReturn(null);
EasyMock.replay(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
final AWSEndpointConfig schemelessEndpoint = MAPPER.readValue(
@@ -542,6 +548,8 @@ public class S3InputSourceTest extends
InitializedNullHandlingTest
EasyMock.reset(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getS3StorageConfig()).andStubReturn(S3_STORAGE_CONFIG);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleConfig()).andStubReturn(null);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleEndpointConfig()).andStubReturn(null);
EasyMock.replay(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
final S3InputSource withPrefixes = new S3InputSource(
SERVICE,
@@ -583,6 +591,8 @@ public class S3InputSourceTest extends
InitializedNullHandlingTest
EasyMock.replay(mockS3StorageConfig);
EasyMock.reset(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getS3StorageConfig()).andStubReturn(mockS3StorageConfig);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleConfig()).andStubReturn(null);
+
EasyMock.expect(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER.getIntermediateRoleEndpointConfig()).andStubReturn(null);
EasyMock.replay(SERVER_SIDE_ENCRYPTING_AMAZON_S3_BUILDER);
final S3InputSource withPrefixes = new S3InputSource(
SERVICE,
diff --git
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3IntermediateRoleConfigTest.java
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3IntermediateRoleConfigTest.java
new file mode 100644
index 00000000000..b3d994ed2f6
--- /dev/null
+++
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3IntermediateRoleConfigTest.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.inject.Guice;
+import com.google.inject.Injector;
+import com.google.inject.Scopes;
+import com.google.inject.name.Names;
+import jakarta.validation.Validation;
+import jakarta.validation.Validator;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.JsonConfigurator;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Properties;
+import java.util.UUID;
+
+public class S3IntermediateRoleConfigTest
+{
+ private static final String PROPERTY_PREFIX = UUID.randomUUID().toString();
+ private static final String INTERMEDIATE_ROLE_ARN =
"arn:aws:iam::123456789012:role/intermediate";
+
+ private final Properties properties = new Properties();
+
+ @BeforeEach
+ public void setUp()
+ {
+ properties.clear();
+ }
+
+ @AfterEach
+ public void tearDown()
+ {
+ properties.clear();
+ }
+
+ private Injector createInjector()
+ {
+ return Guice.createInjector(
+ binder -> {
+
binder.bindConstant().annotatedWith(Names.named("serviceName")).to("druid/test/s3");
+
binder.bindConstant().annotatedWith(Names.named("servicePort")).to(0);
+
binder.bindConstant().annotatedWith(Names.named("tlsServicePort")).to(-1);
+
binder.bind(Validator.class).toInstance(Validation.buildDefaultValidatorFactory().getValidator());
+ binder.bindScope(LazySingleton.class, Scopes.SINGLETON);
+ // The mapper matters: see
testSiblingPropertiesUnderTheSamePrefixAreIgnored.
+ binder.bind(ObjectMapper.class).toInstance(new
DefaultObjectMapper());
+ binder.bind(JsonConfigurator.class).in(LazySingleton.class);
+ binder.bind(Properties.class).toInstance(properties);
+ JsonConfigProvider.bind(binder, PROPERTY_PREFIX,
S3IntermediateRoleConfig.class);
+ }
+ );
+ }
+
+ @Test
+ public void testIntermediateAssumeRoleArnIsUnsetByDefault()
+ {
+ final S3IntermediateRoleConfig config =
createInjector().getInstance(S3IntermediateRoleConfig.class);
+ Assertions.assertNull(config.getIntermediateAssumeRoleArn());
+ }
+
+ @Test
+ public void testIntermediateAssumeRoleArnIsBoundFromProperties()
+ {
+ properties.put(PROPERTY_PREFIX + ".intermediateAssumeRoleArn",
INTERMEDIATE_ROLE_ARN);
+
+ final S3IntermediateRoleConfig config =
createInjector().getInstance(S3IntermediateRoleConfig.class);
+ Assertions.assertEquals(INTERMEDIATE_ROLE_ARN,
config.getIntermediateAssumeRoleArn());
+ }
+
+ @Test
+ public void testSiblingPropertiesUnderTheSamePrefixAreIgnored()
+ {
+ properties.put(PROPERTY_PREFIX + ".intermediateAssumeRoleArn",
INTERMEDIATE_ROLE_ARN);
+ properties.put(PROPERTY_PREFIX + ".accessKey", "someAccessKey");
+ properties.put(PROPERTY_PREFIX + ".retryMode", "adaptive");
+
+ final S3IntermediateRoleConfig config =
createInjector().getInstance(S3IntermediateRoleConfig.class);
+ Assertions.assertEquals(INTERMEDIATE_ROLE_ARN,
config.getIntermediateAssumeRoleArn());
+ }
+}
diff --git
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3MultipartUploadTest.java
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3MultipartUploadTest.java
index ce3979db5a8..69d62919d10 100644
---
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3MultipartUploadTest.java
+++
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3MultipartUploadTest.java
@@ -111,6 +111,8 @@ public class ServerSideEncryptingAmazonS3MultipartUploadTest
endpointConfig,
clientConfig,
null,
+ null,
+ null,
null
).build();
diff --git
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3Test.java
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3Test.java
index 6f16924c524..b4430749cc5 100644
---
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3Test.java
+++
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3Test.java
@@ -20,6 +20,8 @@
package org.apache.druid.storage.s3;
import org.apache.druid.common.aws.AWSClientConfig;
+import org.apache.druid.common.aws.AWSEndpointConfig;
+import org.apache.druid.data.input.s3.S3InputSourceConfig;
import org.easymock.Capture;
import org.easymock.EasyMock;
import org.junit.jupiter.api.Assertions;
@@ -28,6 +30,7 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.retries.api.RetryStrategy;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
@@ -35,6 +38,12 @@ import software.amazon.awssdk.services.s3.model.Grant;
import software.amazon.awssdk.services.s3.model.Grantee;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+import software.amazon.awssdk.services.sts.StsClient;
+import
software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
+import software.amazon.awssdk.services.sts.auth.StsCredentialsProvider;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+import software.amazon.awssdk.services.sts.model.Credentials;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.CompletedFileUpload;
import software.amazon.awssdk.transfer.s3.model.FileUpload;
@@ -43,6 +52,9 @@ import
software.amazon.awssdk.transfer.s3.model.UploadFileRequest;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
@@ -50,6 +62,27 @@ import java.util.concurrent.CompletableFuture;
public class ServerSideEncryptingAmazonS3Test
{
+ private static final String INTERMEDIATE_ROLE_ARN =
"arn:aws:iam::123456789012:role/intermediate";
+ private static final String TARGET_ROLE_ARN =
"arn:aws:iam::123456789012:role/target";
+
+ private static final AWSEndpointConfig REGION_CONFIG = new
AWSEndpointConfig()
+ {
+ @Override
+ public String getSigningRegion()
+ {
+ return "us-east-1";
+ }
+ };
+
+ private static final AWSEndpointConfig INTERMEDIATE_REGION_CONFIG = new
AWSEndpointConfig()
+ {
+ @Override
+ public String getSigningRegion()
+ {
+ return "eu-south-1";
+ }
+ };
+
@TempDir
public File temporaryFolder;
@@ -282,6 +315,8 @@ public class ServerSideEncryptingAmazonS3Test
null,
clientConfig,
null,
+ null,
+ null,
null
);
@@ -289,6 +324,128 @@ public class ServerSideEncryptingAmazonS3Test
Assertions.assertNotSame(issued.get(0), issued.get(1));
}
+ @Test
+ public void testSpecCredentialsAssumeTheRoleTheSpecNamesDirectly() throws
Exception
+ {
+ final ServerSideEncryptingAmazonS3 s3 =
ServerSideEncryptingAmazonS3.builder(
+ AnonymousCredentialsProvider.create(),
+ new S3StorageConfig(new NoopServerSideEncryption(), new
S3TransferConfig()),
+ null,
+ REGION_CONFIG,
+ new AWSClientConfig(),
+ new S3InputSourceConfig(() -> "specAccessKey", () -> "specSecretKey",
TARGET_ROLE_ARN, null, null),
+ null,
+ new S3IntermediateRoleConfig(INTERMEDIATE_ROLE_ARN),
+ null
+ ).build();
+
+ final StsAssumeRoleCredentialsProvider target = assumeRoleProvider(
+ s3.getS3Client().serviceClientConfiguration().credentialsProvider(),
+ "the S3 client should assume a role"
+ );
+ final Field stsClientField =
StsCredentialsProvider.class.getDeclaredField("stsClient");
+ stsClientField.setAccessible(true);
+ final StsClient targetStsClient = (StsClient) stsClientField.get(target);
+
+ try {
+ Assertions.assertEquals(TARGET_ROLE_ARN, roleArnRequestedBy(target));
+ Assertions.assertFalse(
+ targetStsClient.serviceClientConfiguration().credentialsProvider()
instanceof StsAssumeRoleCredentialsProvider,
+ "the spec's own credentials assume the spec's role without the
intermediate role"
+ );
+ Assertions.assertEquals(
+ "specAccessKey",
+
targetStsClient.serviceClientConfiguration().credentialsProvider().resolveIdentity().join().accessKeyId()
+ );
+ }
+ finally {
+ target.close();
+ }
+ }
+
+ @Test
+ public void testIntermediateRoleIsAssumedBeforeTheRoleTheSpecNames() throws
Exception
+ {
+ final ServerSideEncryptingAmazonS3 s3 =
ServerSideEncryptingAmazonS3.builder(
+ AnonymousCredentialsProvider.create(),
+ new S3StorageConfig(new NoopServerSideEncryption(), new
S3TransferConfig()),
+ null,
+ REGION_CONFIG,
+ new AWSClientConfig(),
+ new S3InputSourceConfig(null, null, TARGET_ROLE_ARN, null, null),
+ null,
+ new S3IntermediateRoleConfig(INTERMEDIATE_ROLE_ARN),
+ INTERMEDIATE_REGION_CONFIG
+ ).build();
+
+ final StsAssumeRoleCredentialsProvider target = assumeRoleProvider(
+ s3.getS3Client().serviceClientConfiguration().credentialsProvider(),
+ "the S3 client should assume a role"
+ );
+ // The STS client the target provider calls out with carries the
credentials of the previous link in the chain.
+ final Field stsClientField =
StsCredentialsProvider.class.getDeclaredField("stsClient");
+ stsClientField.setAccessible(true);
+ final StsClient targetStsClient = (StsClient) stsClientField.get(target);
+
+ final StsAssumeRoleCredentialsProvider intermediate = assumeRoleProvider(
+ targetStsClient.serviceClientConfiguration().credentialsProvider(),
+ "the STS client should itself assume the intermediate role"
+ );
+
+ try {
+ Assertions.assertEquals(
+ TARGET_ROLE_ARN,
+ roleArnRequestedBy(target),
+ "the S3 client assumes the role the spec named"
+ );
+ Assertions.assertEquals(
+ INTERMEDIATE_ROLE_ARN,
+ roleArnRequestedBy(intermediate),
+ "and it does so with credentials from assuming the intermediate role
first"
+ );
+ Assertions.assertEquals(Region.US_EAST_1,
targetStsClient.serviceClientConfiguration().region());
+ Assertions.assertEquals(
+ Region.EU_SOUTH_1,
+ ((StsClient)
stsClientField.get(intermediate)).serviceClientConfiguration().region()
+ );
+ }
+ finally {
+ target.close();
+ intermediate.close();
+ }
+ }
+
+ private static StsAssumeRoleCredentialsProvider assumeRoleProvider(Object
provider, String message)
+ {
+ return Assertions.assertInstanceOf(StsAssumeRoleCredentialsProvider.class,
provider, message);
+ }
+
+ private static String roleArnRequestedBy(StsAssumeRoleCredentialsProvider
provider) throws Exception
+ {
+ final Capture<AssumeRoleRequest> requested = EasyMock.newCapture();
+ final StsClient stsClient = EasyMock.createMock(StsClient.class);
+ final AssumeRoleResponse response = AssumeRoleResponse.builder()
+ .credentials(
+ Credentials.builder()
+ .accessKeyId("accessKey")
+ .secretAccessKey("secretKey")
+ .sessionToken("sessionToken")
+ .expiration(Instant.now().plus(Duration.ofHours(1)))
+ .build()
+ )
+ .build();
+
EasyMock.expect(stsClient.assumeRole(EasyMock.capture(requested))).andReturn(response);
+ EasyMock.replay(stsClient);
+
+ final Method getUpdatedCredentials =
+
StsAssumeRoleCredentialsProvider.class.getDeclaredMethod("getUpdatedCredentials",
StsClient.class);
+ getUpdatedCredentials.setAccessible(true);
+ getUpdatedCredentials.invoke(provider, stsClient);
+
+ EasyMock.verify(stsClient);
+ return requested.getValue().roleArn();
+ }
+
@Test
public void testBuilderWithAsyncClient() throws NoSuchFieldException,
IllegalAccessException
{
diff --git
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/TestAWSCredentialsProvider.java
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/TestAWSCredentialsProvider.java
index cc362f6ba8d..1ca307a7e9a 100644
---
a/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/TestAWSCredentialsProvider.java
+++
b/extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/TestAWSCredentialsProvider.java
@@ -66,7 +66,8 @@ public class TestAWSCredentialsProvider
new AWSProxyConfig(),
new AWSEndpointConfig(),
new AWSClientConfig(),
- new S3StorageConfig(new NoopServerSideEncryption(), null)
+ new S3StorageConfig(new NoopServerSideEncryption(), null),
+ new S3IntermediateRoleConfig()
);
s3Module.getAmazonS3Client(
@@ -101,7 +102,8 @@ public class TestAWSCredentialsProvider
new AWSProxyConfig(),
new AWSEndpointConfig(),
new AWSClientConfig(),
- new S3StorageConfig(new NoopServerSideEncryption(), null)
+ new S3StorageConfig(new NoopServerSideEncryption(), null),
+ new S3IntermediateRoleConfig()
);
s3Module.getAmazonS3Client(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
