This is the gdal setup. None of the AWS keys are set excep region (could
that be an issue). Where would I expect to see the logging - none is
showing up in our logs. Do we need to configure the output to our log?
public GdalClient(IConfigurationStore configStore, ILoggerFactory
loggerFactory)
{
_log = loggerFactory.CreateLogger<GdalClient>();
GdalBase.ConfigureAll();
Gdal.AllRegister();
//For debugging s3 access problems
Gdal.SetConfigOption("CPL_CURL_VERBOSE", "YES");
Gdal.SetConfigOption("CPL_DEBUG", "YES");
Gdal.SetConfigOption("AWS_REGION", "us-west-2");
_log.LogDebug($"#{nameof(GdalClient)} AWS_PROFILE: {
Gdal.GetConfigOption("AWS_PROFILE", "NotFound")} " +
$"AWS_ACCESS_KEY_ID: {
Gdal.GetConfigOption("AWS_ACCESS_KEY_ID", "NotFound")} " +
$"AWS_SECRET_ACCESS_KEY: {
Gdal.GetConfigOption("AWS_SECRET_ACCESS_KEY", "NotFound")} " +
$"AWS_SESSION_TOKEN: {
Gdal.GetConfigOption("AWS_SESSION_TOKEN", "NotFound")} " +
$"AWS_NO_SIGN_REQUEST: {
Gdal.GetConfigOption("AWS_NO_SIGN_REQUEST", "NotFound")} " +
$"AWS_DEFAULT_PROFILE: {
Gdal.GetConfigOption("AWS_DEFAULT_PROFILE", "NotFound")} ");
_awsBucketName =
configStore.GetValueString("AWS_DESIGNIMPORT_BUCKET_NAME");
}
and this is the open which times out:
public string GetMetadata(string fileName, bool s3=true)
{
var path = s3 ? $"/vsis3/{_awsBucketName}/{fileName}" : fileName;
var dataset = Gdal.Open(path, Access.GA_ReadOnly);
.........
On Thu, Apr 15, 2021 at 9:11 AM Patrick Young <
patrick.mckendree.yo...@gmail.com> wrote:
> That should be the behavior, and I use it all the time so I am confident
> it works...
>
> Is it possible you have set any of the AWS_* variables in your
> EC2/container environment? I think that could spoil it for you.
>
> You can set CPL_CURL_VERBOSE to YES and get an idea of the network
> requests GDAL is doing. If things are just slow, you might need to set
> GDAL_DISABLE_READDIR_ON_OPEN=YES
> and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif as described here:
>
> https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF
>
> On Wed, Apr 14, 2021 at 3:01 PM Jeannie May <jeannie_...@trimble.com>
> wrote:
>
>> Further to this discussion, I guess we had hoped that GDal would pick up
>> the EC2 instance profile credentials as the s3Client upload does as per:
>>
>> https://gdal.org/user/virtual_file_systems.html#vsis3
>> 5. If none of the above method succeeds, instance profile credentials
>> will be retrieved when GDAL is used on EC2 instances.
>>
>> On Thu, Apr 15, 2021 at 8:58 AM Jeannie May <jeannie_...@trimble.com>
>> wrote:
>>
>>> Thank you Patrick for your prompt reply.
>>>
>>> Our app runs on an EC2 instance and has no user context. It uses the
>>> s3Client SDK upload, utilizing the existing IAM role/policy already setup.
>>>
>>> Are you saying here that I need to as a 1-time process generate a secret
>>> and access key for our existing policy/role, store it in environment
>>> variables (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID only?) and the
>>> gdal SDK will use that?
>>>
>>> Jeannie May
>>>
>>>
>>>
>>> Jeannie M
>>>
>>>
>>>
>>> ay <jeannie_...@trimble.com>
>>> Tue, Apr 13, 2:44 PM (2 days ago)
>>> Reply
>>> to gdal-dev
>>> I'm new to using Gdal. How do I configure gdal to use an IAM role,
>>> rather than defining an aws-Profile?
>>>
>>> I get a timeout doing a gdal.Open() on a tiff file on S3 using vsis3,
>>> while trying to use an IAM role.
>>>
>>> I'm using MaxRev.Gdal.Core 3.2.0.250. Netcore 3.1 c#, running in a Linux
>>> container.
>>>
>>> Note that defining an AWS_Profile etc works ok, but I need to use IAM
>>> roles. Prior to trying to open the file with gdal, I successfully upload
>>> using s3Client (which uses the IAM role), so it's something specific to the
>>> Gdal.Open().
>>>
>>> using GetConfigOption() I can see that none of the following are set
>>> AWS_PROFILE; AWS_ACCESS_KEY_ID; AWS_SECRET_ACCESS_KEY; AWS_SESSION_TOKEN;
>>> AWS_NO_SIGN_REQUEST; AWS_DEFAULT_PROFILE
>>>
>>>
>>> --
>>> *Jeannie May*
>>> Senior Software Engineer
>>> 11 Birmingham Drive, Christchurch | 963 5305 Office
>>> www.trimble.com
>>>
>>> Connect with us!
>>> Patrick Young <patrick.mckendree.yo...@gmail.com>
>>> Wed, Apr 14, 3:47 AM (1 day ago)
>>> Reply
>>> to me, gdal
>>> See
>>> https://aws.amazon.com/premiumsupport/knowledge-center/iam-assume-role-cli/
>>> ,
>>> they describe how to assume a role and set the usual AWS_*
>>> environment variables that GDAL should pick up.
>>>
>>> There's discussion on vsis3 related stuff (e.g. authentication) here:
>>> https://gdal.org/user/virtual_file_systems.html#vsis3
>>>
>>> P
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> *Jeannie May*
>> Senior Software Engineer
>> 11 Birmingham Drive, Christchurch | 963 5305 Office
>> www.trimble.com
>>
>> Connect with us!
>>
>
--
*Jeannie May*
Senior Software Engineer
11 Birmingham Drive, Christchurch | 963 5305 Office
www.trimble.com
Connect with us!
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
