Hi Darren You can do this by setting `GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR` or by narrowing the allowed extension with "CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif"
Kyle is working on some docs that explains those env in TiTiler: - https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#gdal_disable_readdir_on_open <https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#gdal_disable_readdir_on_open> - https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#cpl_vsil_curl_allowed_extensions <https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#cpl_vsil_curl_allowed_extensions> Vincent > On Mar 3, 2021, at 6:56 PM, Darren Weber <darren.we...@jupiterintel.com> > wrote: > > In https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225 > <https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225> I've > noted how s3-COG reads can use retries to handle s3 rate throttling (503) by > enabling the `GDAL_HTTP_MAX_RETRY`. Our dataset on s3 only contains *.tif > files and we need to prevent GDAL from issuing any GET/HEAD requests to try > to read any other files. Is there an env-var for this? > > The `GDAL_DISABLE_READDIR_ON_OPEN` is `False` so that GDAL can read the s3 > prefix to list objects in one GET request. From that read, it should have > already determined that there are only *.tif files and no *.aux* files > available. When it gets a 503 rate throttle response, however, it seems to > auto-try to read some supplementary files, e.g. one log contains messages > that indicate GDAL is trying to read an .aux file that does not exist and we > want to prevent GDAL from issuing any GET or HEAD requests for anything other > than *.tif files (objects). > > [WARNING] 2021-03-03T19:37:40.312Z CPLE_AppDefined in HTTP error code: 503 - > https://a-bucket.s3.amazonaws.com/geotiff.aux > <https://a-bucket.s3.amazonaws.com/geotiff.aux>. Retrying again in 0.5 secs > [WARNING] 2021-03-03T19:37:40.795Z CPLE_AppDefined in HTTP error code: 503 - > https://a-bucket.s3.amazonaws.com/geotiff.aux > <https://a-bucket.s3.amazonaws.com/geotiff.aux>. Retrying again in 1.0 secs > [WARNING] 2021-03-03T19:37:41.796Z CPLE_AppDefined in HTTP error code: 503 - > https://a-bucket.s3.amazonaws.com/geotiff.aux > <https://a-bucket.s3.amazonaws.com/geotiff.aux>. Retrying again in 2.2 secs > [WARNING] 2021-03-03T19:37:44.290Z CPLE_AppDefined in HTTP response code on > https://a-bucket.s3.amazonaws.com/geotiff.aux > <https://a-bucket.s3.amazonaws.com/geotiff.aux>: 503 > 2021-03-03 19:37:44,155 | INFO | extract_raster_points:1036 | > s3://a-bucket/geotiff.tif extract cells (1)... > 2021-03-03 19:37:44,347 | INFO | extract_raster_points:1050 | > s3://a-bucket/geotiff.tif extract cells (1) done > That log suggests that there is a huge performance hit to retry reading an > .aux file that does not exist and GDAL should not even try to issue any kind > of request for it (i.e. we want to prevent it). > > When running some unit tests with debug curl details available (but no > ability to create 503 responses), it appears that GDAL finds just the file > required and reads only partial reads from that file, e.g. this is a sample > of GET requests to read from one .tif s3-COG: > > $ grep 'GET' tmp.log > > GET /?delimiter=%2F&prefix=unit_tests%2Fgis%2F HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > GET /unit_tests/gis/cea_blocks.tif HTTP/1.1 > > TIA, Darren > > > -- > Darren Weber, PhD > Senior Software Engineer > Jupiter Intelligence > > > "Predicting risk in a changing climate" > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
