Thanks Michael and Even GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR works perfectly for me, in gdalinfo and QGIS. Great.
Even. No, I was not setting GDAL_DISABLE_READDIR_ON_OPEN at all. I think I do not have directory listing permission in that folder/bucket (that is why it returns a 403, and not a 404, right?) Cheers, Javier. On Sat, 27 Apr 2024 at 13:48, Even Rouault <even.roua...@spatialys.com> wrote: > yes, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the trick to both disable > directory listing and preventing individual file probing. > > Javier, do you have GDAL_DISABLE_READDIR_ON_OPEN=YES also set? Otherwise > if GDAL_DISABLE_READDIR_ON_OPEN is not set, GDAL will normally issue a S3 > directory listing call, which should prevent later individual file probing > (if we have the list of files in the directory, we just need to check the > list, and don't need to do individual open()/GET attempts). But that > director listing is sometimes still undesirable if doing it in a folder > with a large number of files hence GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR. > Or maybe you don't have directory listing permission in that folder/bucket? > > Related to that, Howard filed https://github.com/OSGeo/gdal/issues/9443 > recently. but it is not that simple to find a different default behavior > that doesn't result in functional regressions. > > I was wondering if a solution, restricted to COG, couldn't be to put a > hint in the so-called "header ghost area" ( > https://gdal.org/drivers/raster/cog.html#header-ghost-area), like > SIDECAR_FILES=NO, that could be used by the GTiff driver. Not sure if it is > the right way to address the issue however (and would work only for new > files of course, and probably only on explicit user choice at file > creation, because there could be scenarios where a user would want to > provide side car files) > Le 27/04/2024 à 13:12, Michael Smith via gdal-dev a écrit : > > Javier, > > > > You can control with the GDAL_DISABLE_READDIR_ON_OPEN configuration > parameter (https://gdal.org/user/configoptions.html). Typically, I set > GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which will disable this. > > > > Mike > > > > > > -- > > Michael Smith > > Remote Sensing/GIS Center > > US Army Corps of Engineers > > > > > > *From: *gdal-dev <gdal-dev-boun...@lists.osgeo.org> > <gdal-dev-boun...@lists.osgeo.org> on behalf of Javier Jimenez Shaw via > gdal-dev <gdal-dev@lists.osgeo.org> <gdal-dev@lists.osgeo.org> > *Reply-To: *Javier Jimenez Shaw <j...@jimenezshaw.com> <j...@jimenezshaw.com> > *Date: *Saturday, April 27, 2024 at 7:01 AM > *To: *gdal dev <gdal-dev@lists.osgeo.org> <gdal-dev@lists.osgeo.org> > *Subject: *[gdal-dev] don't search side car files on s3 with gdalinfo > > > > Hi > > > > I am accessing a big file in S3. All works fine (when I learned how to set > the authentication environment variables), but gdalinfo is checking for a > lot of files that do not exist. I know there is only a GeoTIFF - COG (no > side car files) > > > > However gdalinfo is trying to find several side car files, that are not > there. S3 returns a 403. It takes its time to search for all those files. > > > > Is there a way to tell to gdalinfo that it should not check for those > files? > > > > Bonus: QGIS is also taking time to open it (just using 'qgis' instead of > 'gdalinfo' in the command line). I think it is for the same reason. It > there a hack there as well? > > > > Thanks. > > > > $ AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... > gdalinfo /vsis3/blah/folder/ortho.tiff > Driver: GTiff/GeoTIFF > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.aux: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.AUX: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.aux: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.msk: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.XML: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.xml: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.IMD: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.imd: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.RPB: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.rpb: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.PVL: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.pvl: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_rpc.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_metadata.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_MTL.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/METADATA.DIM: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/metadata.dim: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_metadata.xml: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.RPC: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.rpc: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.pass: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/ortho.PASS: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/summary.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/SUMMARY.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/HDRdisplay.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/HDRho_display.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/HDRho_display.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/RPCdisplay.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/RPCho_display.txt: 403 > Warning 1: HTTP response code on > https://blah.s3.amazonaws.com/folder/RPCho_display.TXT: 403 > Files: /vsis3/blah/folder/ortho.tiff > > ... > > > > .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__ > > _______________________________________________ gdal-dev mailing list > gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev > > _______________________________________________ > gdal-dev mailing > listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
