[gdal-dev] (TileDB related question for you Norman/Sean) Re: [EXTERNAL] s3 request example

[Even Rouault via gdal-dev]

I've given this a try with my QGIS dev env and I can also reproduce, and 
specifically with the TileDB driver available

The stacktrace is as following:
Thread 1 "qgis" received signal SIGINT, Interrupt.
0x00007f81cc566bcf in __GI___poll (fds=0x7fffc613b040, nfds=2, 
timeout=800) at ../sysdeps/unix/sysv/linux/poll.c:29
29    ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) bt
#0  0x00007f81cc566bcf in __GI___poll (fds=0x7fffc613b040, nfds=2, 
timeout=800) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f809b5fa76a in Curl_poll () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#2  0x00007f809b5f3684 in Curl_multi_wait.part.0 () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#3  0x00007f809b5f3c06 in curl_multi_poll () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#4  0x00007f809b5dfbb0 in curl_easy_perform () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#5  0x00007f809b3d822f in 
Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> 
const&, Aws::Utils::RateLimits::RateLimiterInterface*, 
Aws::Utils::RateLimits::RateLimiterInterface*) const () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#6  0x00007f809b3b0788 in 
Aws::Internal::AWSHttpResourceClient::GetResourceWithAWSWebServiceResult(std::shared_ptr<Aws::Http::HttpRequest> 
const&) const ()
    at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#7  0x00007f809b39c627 in 
Aws::Internal::EC2MetadataClient::GetCurrentRegion() const () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#8  0x00007f809b3a2b8b in 
Aws::Client::ClientConfiguration::ClientConfiguration() () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#9  0x00007f809aae98dd in Aws::Client::ClientConfiguration* 
tiledb::common::tiledb_new<Aws::Client::ClientConfiguration>(std::string 
const&) () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#10 0x00007f809aace812 in tiledb::sm::S3::init_client() const () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#11 0x00007f809aad3d98 in tiledb::sm::S3::is_dir(tiledb::sm::URI const&, 
bool*) const () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#12 0x00007f809aaf745d in tiledb::sm::VFS::is_dir(tiledb::sm::URI 
const&, bool*) const () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#13 0x00007f809b14cf86 in 
tiledb::sm::StorageManagerCanonical::is_array(tiledb::sm::URI const&) 
const () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#14 0x00007f809b14d1c7 in 
tiledb::sm::StorageManagerCanonical::object_type(tiledb::sm::URI const&, 
tiledb::sm::ObjectType*) const () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#15 0x00007f809a9eb479 in tiledb_object_type () at 
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#16 0x00007f80a1a16882 in tiledb::Object::object(tiledb::Context const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&)
    (ctx=..., 
uri="s3://prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff") 
at /home/even/tiledb/tiledb-2.15.2/include/tiledb/object.h:175
#17 0x00007f80a1a1368b in TileDBDataset::Identify(GDALOpenInfo*) 
(poOpenInfo=0x7fffc613ca20) at 
/home/even/gdal/gdal/frmts/tiledb/tiledbcommon.cpp:157
#18 0x00007f80a1a13977 in TileDBDataset::Open(GDALOpenInfo*) 
(poOpenInfo=0x7fffc613ca20) at 
/home/even/gdal/gdal/frmts/tiledb/tiledbcommon.cpp:208
#19 0x00007f81d0430525 in GDALDriver::Open(GDALOpenInfo*, bool) 
(this=0x562cf5138f90, poOpenInfo=0x7fffc613ca20, bSetOpenOptions=false) 
at /home/even/gdal/gdal/gcore/gdaldriver.cpp:117
#20 0x00007f81d044a450 in GDALOpenEx(char const*, unsigned int, char 
const* const*, char const* const*, char const* const*)
    (pszFilename=0x562cf8e7e378 
"/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff", 
nOpenFlags=4, papszAllowedDrivers=0x0, papszOpenOptions=0x562cfb256920, 
papszSiblingFiles=0x0) at /home/even/gdal/gdal/gcore/gdaldataset.cpp:3774
#21 0x00007f81d6b48489 in QgsOgrProviderUtils::GDALOpenWrapper(char 
const*, bool, char**, void**)
    (pszPath=0x562cf8e7e378 
"/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff", 
bUpdate=false, papszOpenOptionsIn=0x562cfb2533f0, phDriver=0x0) at 
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1067
#22 0x00007f81d6b4ab7c in OpenHelper(QString const&, bool, QStringList 
const&) (dsName=..., updateMode=false, options=...) at 
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1460
#23 0x00007f81d6b4b861 in QgsOgrProviderUtils::getLayer(QString const&, 
bool, QStringList const&, int, QString&, bool)
    (dsName=..., updateMode=false, options=..., layerIndex=0, 
errCause=..., checkModificationDateAgainstCache=true) at 
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1603
#24 0x00007f81d6b31f80 in QgsOgrProviderMetadata::querySublayers(QString 
const&, QFlags<Qgis::SublayerQueryFlag>, QgsFeedback*) const 
(this=0x562cf50d8150, u=..., flags=..., feedback=0x0)
    at 
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrprovidermetadata.cpp:970
#25 0x00007f81d6a471d7 in QgsProviderRegistry::querySublayers(QString 
const&, QFlags<Qgis::SublayerQueryFlag>, QgsFeedback*) const 
(this=0x562cf50d31e0, uri=..., flags=..., feedback=0x0)
    at /home/even/qgis/qgis/src/core/providers/qgsproviderregistry.cpp:1114
#26 0x00007f81de8ebeed in QgsAppLayerHandling::openLayer(QString const&, 
bool&, bool, bool, bool)
    (fileName=..., ok=@0x7fffc613d760: false, allowInteractive=false, 
suppressBulkLayerPostProcessing=false, addToLegend=true) at 
/home/even/qgis/qgis/src/app/layers/qgsapplayerhandling.cpp:960
#27 0x0000562cf401167e in main(int, char**) (argc=2, 
argv=0x7fffc613e1c8) at /home/even/qgis/qgis/src/app/main.cpp:1596

So, to make it short, this is when QGIS tries to open the file as a 
vector one (it must tries to open both as raster and vector). And the 
TileDB driver isn't currently smart enough to immediately reject that a 
".tiff" file is not a vector TileDB dataset.  This is not a problem when 
opening as raster dataset because the GeoTIFF driver would already have 
identified the file.

@Norman: only libtiledb can effectively guess if a *directory* contains 
a TileDB dataset or not. But is a *file* a valid URI for a TileDB 
dataset? Because at the time the Identify() method of the TileDB driver 
is hit, GDAL knows already if /vsis3/something is a file or a directory, 
and thus we could immediately bail out if it is a file and a file can't 
be a TileDB dataset.

Even


Le 13/09/2024 à 18:30, thomas bonfort via gdal-dev a écrit :
I also see that you have a tiledb driver registration happening only 
on your conda version, exactly at the time of your lag. you should 
also try to remove/rename your gdal_tiledb.so plugin file to check the 
issue isn't coming from there.

Le ven. 13 sept. 2024, 18:22, thomas bonfort 
<thomas.bonf...@gmail.com> a écrit :

    Javier,
    3.9 is probably doing parallel reads, which will cause a
    noticeable lag in requests on a slow connection (the overall
    processing time should not be slower anyhow). There's a gtiff open
    option documented that can disable this behavior, that you can try
    to use to see if it changes your timings. If that is the effective
    cause, that would mean there is an issue with this parallel
    implementation.
    TB

    Le ven. 13 sept. 2024, 18:15, Javier Jimenez Shaw via gdal-dev
    <gdal-dev@lists.osgeo.org> a écrit :

        Thanks David for the suggestion. Very interesting data.

        Unfortunately the issue I have is not reproduced with that
        file. So it may be due to the authentication (your suggestion
        does not need authentication).

        Some context: I am opening a COG file from S3 in QGIS, setting
        the url in the command line. Something like
        qgis /vsis3/blahblabla

        (more on that later)

        If I run it in my Ubuntu 22.04 (that has GDAL 3.4.1), with
        QGIS 3.38, no problem
        If I run with conda with QGIS 3.38 (that has GDAL 3.9.1), it
        is noticeable slower.

        So I think it is due to the version of GDAL. If you think that
        I am wrong and I should ask directly to QGIS, it is ok.

        This is a bit problematic to reproduce because the
        authorization expires in 12 hours. So I post here the command
        I produced with a python script. If it is needed, I can share
        the python script that is  getting the authentication. It is
        about 100 lines of code.

        Enabling the logs I see there a gap of 6 seconds during the
        opening with conda, that is not there with the QGIS from
        Ubuntu that uses GDAL 3.4.1

        This is the log from conda
        The gap is between
        Fri Sep 13 17:56:26 2024].9370
        and
        [Fri Sep 13 17:56:33 2024].4445

        CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py
        --project-id 1969386 --share-token
        09c3eada-1b3b-46eb-b5de-8bd51635a5b9
        Session expires at 2024-09-14T03:56:22Z
        AWS_ACCESS_KEY_ID=ASIATOCJLBKS2RHGCAHB
        AWS_SECRET_ACCESS_KEY=nNT8fTH/cGiyn4NVubp5/1LroyZ69johhXYxkCK1
        
AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDI1G/49ov88aGibrOCLXAilDbBuFQ/ydJF4AZ50/lYmWYmNDjM9H7XbbioyXiLQzwTqVRN1TLG6T29Anklkw47MKxdKHDF/O85r9gWJmCAqYTKalA38VM1qbixsjSXAzR7W5p1v1pgRUKtN4J03NrZOH2S6gbo4OKv+UUkPe/Xq4s8w0vEVNJLhvVx6dvpJMv7Jp9UxvLA5a+nrBKUqJujujqJnRXAKKfnpJbTDvhCuIeTxaFFt50e5UpLjSwqXS/a/AaEMJXRSsSLbKFTW/ALSCHfPv3HCStnue5CFpzVk4OhQbQ+dZyXUQp5wxQTtpr0hIh0IyB20gsoZNa0lGvFuXnzqDCH2X2mXwzDcyZk2IOX34nq+Wtnk8qYdjJxEAG0tnlYsMjmmXECnbOxl6gtNJ5hoehgak6cSfs7/rMi4jOI838tzsWWyDJeACj9qfQyfGtpk23x8Zt6KuqUjNZ1/z+/T8LwoopsGRtwYyKQbJS8pn8IGKPLtHDVxRx09yBRuqgYVFzSqpnr3T1vwWYHXiEF0sVJBx
        GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
        /home/jshaw/anaconda3/envs/qgis_stable/bin/qgis
        
/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
        [Fri Sep 13 17:56:26 2024].3948, 0.0000: HTTP: libcurl/8.9.0
        OpenSSL/3.3.1 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
        [Fri Sep 13 17:56:26 2024].7731, 0.3783: S3: Switching to
        endpoint s3.eu-central-1.amazonaws.com
        <http://s3.eu-central-1.amazonaws.com>
        [Fri Sep 13 17:56:26 2024].8415, 0.4467: S3: Switching to
        region eu-central-1
        [Fri Sep 13 17:56:26 2024].9230, 0.5282: S3:
        
GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
         response_code=206
        [Fri Sep 13 17:56:26 2024].9235, 0.5288: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x556e6bb915a0) succeeds as GTiff.
        [Fri Sep 13 17:56:26 2024].9240, 0.5292: GTiff: ScanDirectories()
        [Fri Sep 13 17:56:26 2024].9243, 0.5296: GTiff: Opened band mask.
        [Fri Sep 13 17:56:26 2024].9246, 0.5298: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 17:56:26 2024].9249, 0.5301: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 17:56:26 2024].9251, 0.5304: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 17:56:26 2024].9254, 0.5306: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 17:56:26 2024].9257, 0.5309: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 17:56:26 2024].9259, 0.5311: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 17:56:26 2024].9262, 0.5314: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 17:56:26 2024].9264, 0.5316: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 17:56:26 2024].9267, 0.5319: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 17:56:26 2024].9269, 0.5321: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 17:56:26 2024].9272, 0.5324: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 17:56:26 2024].9274, 0.5326: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 17:56:26 2024].9277, 0.5329: GDAL:
        
GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x556e6bb915a0)
        ERROR: Status 2: File
        
/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
        could not be found
        [Fri Sep 13 17:56:26 2024].9370, 0.5422: GDAL: On-demand
        registering
        /home/jshaw/anaconda3/envs/qgis_stable/lib/gdalplugins/gdal_TileDB.so
        using GDALRegister_TileDB.
        [Fri Sep 13 17:56:33 2024].4445, 7.0498: S3:
        
GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
         response_code=206
        [Fri Sep 13 17:56:33 2024].4449, 7.0501: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x556e6b807d00) succeeds as GTiff.
        GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4666, 7.0718: OGRCT:
        Using WebMercator to WGS84 optimization
        GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4667, 7.0719: GDAL:
        Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
        [Fri Sep 13 17:56:33 2024].4667, 7.0719: GTiff: ScanDirectories()
        [Fri Sep 13 17:56:33 2024].4669, 7.0721: GTiff: Opened band mask.
        [Fri Sep 13 17:56:33 2024].4670, 7.0722: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 17:56:33 2024].4671, 7.0723: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 17:56:33 2024].4672, 7.0724: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 17:56:33 2024].4673, 7.0725: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 17:56:33 2024].4674, 7.0726: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 17:56:33 2024].4675, 7.0727: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 17:56:33 2024].4676, 7.0728: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 17:56:33 2024].4677, 7.0729: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 17:56:33 2024].4677, 7.0730: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 17:56:33 2024].4678, 7.0731: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 17:56:33 2024].4679, 7.0732: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 17:56:33 2024].4680, 7.0733: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 17:56:33 2024].5178, 7.1230: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x7f830c08d490) succeeds as GTiff.
        [Fri Sep 13 17:56:33 2024].5180, 7.1232: S3: Downloading
        59179008-59244543
        
(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].5244, 7.1296: GDAL:
        Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
        [Fri Sep 13 17:56:33 2024].5245, 7.1297: GTiff: ScanDirectories()
        [Fri Sep 13 17:56:33 2024].5246, 7.1298: GTiff: Opened band mask.
        [Fri Sep 13 17:56:33 2024].5247, 7.1299: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 17:56:33 2024].5247, 7.1300: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 17:56:33 2024].5248, 7.1300: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 17:56:33 2024].5249, 7.1301: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 17:56:33 2024].5249, 7.1302: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 17:56:33 2024].5250, 7.1302: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 17:56:33 2024].5251, 7.1303: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 17:56:33 2024].5251, 7.1304: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 17:56:33 2024].5252, 7.1304: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 17:56:33 2024].5253, 7.1305: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 17:56:33 2024].5253, 7.1306: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 17:56:33 2024].5254, 7.1306: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 17:56:33 2024].5255, 7.1307: GDAL: Selecting
        overview 1116 x 900
        [Fri Sep 13 17:56:33 2024].5255, 7.1307: S3: Downloading
        16384-32767
        
(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 17:56:33 2024].6116, 7.2168: S3: Got response_code=206
        [Fri Sep 13 17:56:33 2024].6118, 7.2171: S3: Downloading
        180224-589823
        
(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 17:56:33 2024].6955, 7.3008: S3: Got response_code=206
        [Fri Sep 13 17:56:33 2024].6960, 7.3012: GDAL: GDAL_CACHEMAX =
        1591 MB
        [Fri Sep 13 17:56:33 2024].6977, 7.3029: S3: Got response_code=206


        This is the equivalent code with QGIS 3.38 in Ubuntu and GDAL
        3.4.1

        CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py
        --project-id 1969386 --share-token
        09c3eada-1b3b-46eb-b5de-8bd51635a5b9
        Session expires at 2024-09-14T04:00:10Z
        AWS_ACCESS_KEY_ID=ASIATOCJLBKSRCEUBKE6
        AWS_SECRET_ACCESS_KEY=h6NhBf9VwBZoVj5Zd/8H2BnzXQu0l6bFUUv2D8E7
        
AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDF1xO50XL6GBe/358CLXAn3JR/uteJtarwJ/ryn6i4+lcMFEQZ/KV0FZQdKsD1PwAcqoYFY+0GKYSV+FTk81ZszsVzy8N7d2GwByvP9S/L1XOo1FrqJfDpGw3a5eQR7AoeU68EPlD/WTYnM7XBzXimHywDCyue3SOi8oZaSdbMgjBQ1N6j+qQ+EDN7RjD/Zu2CGMTzN9g/Z0FhZsuyt2HRhnOYpsvpUMZ+WQr7pbjdVdwt3PdFHsJ95fhueapJspgQ7OSgmj5VtSR8NTecohLsd2eeGkSpiqVWuBwuAu2cW6LcqmAZGnF0DZIjPWiMIDf/OLczMRCw5MJGeYpsSkkiuM+t9Miv8aN4iCo8vjl6wumOCgPqTgBBWlQdAs2mLxwpF1Bb1UjJ6eiccp17+yJF3ffn8tvxx/+1V3Q++E5Xk+jiB1keJnFcizr7qD6EY+EdDES5sQrWAhM2uR78JjNc0f+hVZEmAoisORtwYyKb0po07ytiiBv+X3j+Y6BsCHR0Es3C3xVvsgkfn1zb1Z2flFWUzwqJUD
        GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR /usr/bin/qgis
        
/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
        [Fri Sep 13 18:00:14 2024].7665, 0.0000: HTTP: libcurl/7.81.0
        OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8
        libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2)
        libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3
        OpenLDAP/2.5.18
        [Fri Sep 13 18:00:14 2024].7666, 0.0001: S3: Downloading
        0-16383
        
(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 18:00:14 2024].8897, 0.1231: S3: Got response_code=400
        [Fri Sep 13 18:00:14 2024].8898, 0.1232: S3: Switching to
        region eu-central-1
        [Fri Sep 13 18:00:14 2024].8899, 0.1234: S3: Downloading
        0-16383
        
(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 18:00:15 2024].0287, 0.2621: S3: Got response_code=206
        [Fri Sep 13 18:00:15 2024].0288, 0.2622: S3:
        
GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
         response_code=206
        [Fri Sep 13 18:00:15 2024].0292, 0.2627: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x560e4cd84920) succeeds as GTiff.
        [Fri Sep 13 18:00:15 2024].0298, 0.2632: GTiff: ScanDirectories()
        [Fri Sep 13 18:00:15 2024].0301, 0.2635: GTiff: Opened band mask.
        [Fri Sep 13 18:00:15 2024].0304, 0.2638: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].0306, 0.2641: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].0309, 0.2643: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].0311, 0.2646: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 18:00:15 2024].0314, 0.2648: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 18:00:15 2024].0316, 0.2651: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 18:00:15 2024].0319, 0.2654: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].0322, 0.2656: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].0324, 0.2658: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].0326, 0.2661: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 18:00:15 2024].0329, 0.2663: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 18:00:15 2024].0331, 0.2665: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 18:00:15 2024].0334, 0.2669: GDAL:
        
GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x560e4cd84920)
        ERROR: Status 2: File
        
/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
        could not be found
        [Fri Sep 13 18:00:15 2024].0357, 0.2691: S3: Downloading
        0-16383
        
(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 18:00:15 2024].0762, 0.3096: S3: Got response_code=206
        [Fri Sep 13 18:00:15 2024].0762, 0.3097: S3:
        
GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
         response_code=206
        [Fri Sep 13 18:00:15 2024].0765, 0.3099: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x560e4d0afb70) succeeds as GTiff.
        GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].0956, 0.3291: GDAL:
        Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
        [Fri Sep 13 18:00:15 2024].0958, 0.3293: GTiff: ScanDirectories()
        [Fri Sep 13 18:00:15 2024].0959, 0.3294: GTiff: Opened band mask.
        [Fri Sep 13 18:00:15 2024].0961, 0.3295: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].0962, 0.3296: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].0962, 0.3297: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].0963, 0.3298: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 18:00:15 2024].0964, 0.3299: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 18:00:15 2024].0965, 0.3300: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 18:00:15 2024].0966, 0.3301: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].0967, 0.3302: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].0968, 0.3303: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].0969, 0.3303: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 18:00:15 2024].0970, 0.3304: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 18:00:15 2024].0971, 0.3305: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 18:00:15 2024].2011, 0.4346: GDAL:
        
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
        this=0x7f0dfc08dfd0) succeeds as GTiff.
        GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].2120, 0.4454: GDAL:
        Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
        [Fri Sep 13 18:00:15 2024].2121, 0.4456: GTiff: ScanDirectories()
        [Fri Sep 13 18:00:15 2024].2122, 0.4457: GTiff: Opened band mask.
        [Fri Sep 13 18:00:15 2024].2123, 0.4457: GTiff: Opened
        8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].2123, 0.4458: GTiff: Opened
        4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].2124, 0.4459: GTiff: Opened
        2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].2125, 0.4459: GTiff: Opened
        1116x900 overview.
        [Fri Sep 13 18:00:15 2024].2125, 0.4460: GTiff: Opened 558x450
        overview.
        [Fri Sep 13 18:00:15 2024].2126, 0.4460: GTiff: Opened 279x225
        overview.
        [Fri Sep 13 18:00:15 2024].2126, 0.4461: GTiff: Opened band
        mask for 8929x7207 overview.
        [Fri Sep 13 18:00:15 2024].2127, 0.4462: GTiff: Opened band
        mask for 4464x3603 overview.
        [Fri Sep 13 18:00:15 2024].2128, 0.4462: GTiff: Opened band
        mask for 2232x1801 overview.
        [Fri Sep 13 18:00:15 2024].2128, 0.4463: GTiff: Opened band
        mask for 1116x900 overview.
        [Fri Sep 13 18:00:15 2024].2129, 0.4463: GTiff: Opened band
        mask for 558x450 overview.
        [Fri Sep 13 18:00:15 2024].2129, 0.4464: GTiff: Opened band
        mask for 279x225 overview.
        [Fri Sep 13 18:00:15 2024].2130, 0.4465: S3: Downloading
        16384-32767
        
(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 18:00:15 2024].3454, 0.5789: S3: Got response_code=206
        [Fri Sep 13 18:00:15 2024].3456, 0.5791: S3: Downloading
        180224-589823
        
(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
        [Fri Sep 13 18:00:15 2024].4272, 0.6606: S3: Got response_code=206
        [Fri Sep 13 18:00:15 2024].4276, 0.6610: GDAL: GDAL_CACHEMAX =
        1591 MB

        What can it be?

        BTW, in both cases it says
        ERROR: Status 2: File
        
/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
        could not be found
        And later it opens it. I don't know why.

        Thank you very much
        Javier.

        On Mon, 9 Sept 2024 at 23:49, Mayer, David P
        <dpma...@usgs.gov> wrote:

            Would ~200 GB be large enough?

            Here is a mosaic of a section of Mars at 6 meters/pixel in
            a public S3 bucket:
            
s3://asc-pds-services/mosaic/Mars/Mars_MRO_CTX_Equi_Mosaics_Robbins/6mpp/MC16_6mpp.16bit.tif

            Cheers,
            David

            -----------
            David Mayer
            Geographer
            Astrogeology Science Center
            US Geological Survey
            dpma...@usgs.gov
            
------------------------------------------------------------------------
            *From:* gdal-dev <gdal-dev-boun...@lists.osgeo.org> on
            behalf of Javier Jimenez Shaw via gdal-dev
            <gdal-dev@lists.osgeo.org>
            *Sent:* Monday, September 9, 2024 13:55
            *To:* gdal dev <gdal-dev@lists.osgeo.org>
            *Subject:* [EXTERNAL] [gdal-dev] s3 request example

            * This email has been received from outside of DOI - Use
            caution before clicking on links, opening attachments, or
            responding. *



            Hi

            I am experiencing a much slower load of a remote COG using
            vsis3 with the QGIS from conda than the QGIS in my Ubuntu
            (that uses GDAL 3.4.1).

            Because both are QGIS 3.38, I think the problem is in the
            version of GDAL.
            I would like to research it more.

            Is there any example of a (big) COG file in Amazon S3 I
            can use to compare them? Unfortunately the credentials for
            the files I am using expire after a few hours, so they are
            useless to report anything.

            Best regards,
            Javier

        _______________________________________________
        gdal-dev mailing list
        gdal-dev@lists.osgeo.org
        https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev