Re: [gdal-dev] vsicurl and --range byte-range

2025-05-08 Thread Joaquim Manuel Freire Luís via gdal-dev
Fantastic Even. Thanks a lot. (and ofc I meant to say grib2, not “gring2”) Joaquim From: Even Rouault Sent: Thursday, May 8, 2025 11:04 PM To: Joaquim Manuel Freire Luís ; gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] vsicurl and --range byte-range Joaquim, You can combine /vsisubfile

Re: [gdal-dev] vsicurl and --range byte-range

2025-05-08 Thread Even Rouault via gdal-dev
Joaquim, You can combine /vsisubfile/ (https://gdal.org/en/stable/user/virtual_file_systems.html#vsisubfile-portions-of-files) and /vsicurl/ /vsisubfile/${offset}_${length}/vsicurl/${url} e.g. $ gdalinfo /vsisubfile/802533_508958,/vsicurl/https://ecmwf-forecasts.s3.eu-central-1.amazonaws.c

[gdal-dev] vsicurl and --range byte-range

2025-05-08 Thread Joaquim Manuel Freire Luís via gdal-dev
Hi, I'm writing a tool to download ECMWF forecast data in gring2 format. But since those files have > 150 bands and most of times I'm only interested in a couple of them, I'm using the only documented mechanism that I found [1], which is the "curl -range start-stop". That works fine but what I

Re: [gdal-dev] vsicurl and HDF5

2023-01-05 Thread Paolo Corti
On Thu, Jan 5, 2023 at 7:14 PM Joe Lee wrote: > > Hi, Paolo! > > > > I tested on mac: > > > > Darwin nene.ad.hdfgroup.org 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6 > 23:31:16 PST 2022; root:xnu-8020.240.14~1/RELEASE_X86_64 x86_64 > > > > I’m using `brew install gdal.` > > > > I just tested

Re: [gdal-dev] vsicurl and HDF5

2023-01-05 Thread Joe Lee via gdal-dev
-dev on behalf of Paolo Corti Date: Thursday, January 5, 2023 at 12:10 PM To: Even Rouault Cc: gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] vsicurl and HDF5 On Thu, Jan 5, 2023 at 6:03 PM Even Rouault wrote: > > Paolo, > > Le 05/01/2023 à 17:56, Paolo Corti a écrit : > >

Re: [gdal-dev] vsicurl and HDF5

2023-01-05 Thread Paolo Corti
On Thu, Jan 5, 2023 at 6:03 PM Even Rouault wrote: > > Paolo, > > Le 05/01/2023 à 17:56, Paolo Corti a écrit : > > Hi all > > > > Do you know if I can read HDF5 files with vsicurl? > > I made a few tests, but I am getting this error, so I was wondering it > > it is even possible: > > yes this is p

Re: [gdal-dev] vsicurl and HDF5

2023-01-05 Thread Joe Lee via gdal-dev
ill=No HDFEOS_GRIDS_VNP_Grid_DNB_Data_Fields_BrightnessTemperature_M12_add_offset=203 … From: gdal-dev on behalf of Paolo Corti Date: Thursday, January 5, 2023 at 10:57 AM To: gdal-dev@lists.osgeo.org Subject: [gdal-dev] vsicurl and HDF5 Hi all Do you know if I can read HDF5 files with vsicurl? I made a few tests, but I am getting this error, so

Re: [gdal-dev] vsicurl and HDF5

2023-01-05 Thread Even Rouault
Paolo, Le 05/01/2023 à 17:56, Paolo Corti a écrit : Hi all Do you know if I can read HDF5 files with vsicurl? I made a few tests, but I am getting this error, so I was wondering it it is even possible: yes this is possible. If you get the error message you mention, it means that the HDF5 dri

[gdal-dev] vsicurl and HDF5

2023-01-05 Thread Paolo Corti
Hi all Do you know if I can read HDF5 files with vsicurl? I made a few tests, but I am getting this error, so I was wondering it it is even possible: gdalinfo /vsicurl/https://./myfile.h5 ERROR 1: CPLCreateUserFaultMapping(): syscall(__NR_userfaultfd) failed: error = 38 gdalinfo failed - un

Re: [gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

2021-05-05 Thread Even Rouault
Jukka, Running with CPL_CURL_VERBOSE=YES shows the following > GET /data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2 HTTP/1.1 Host: ftpprd.ncep.noaa.gov Accept: */* Range: bytes=84819968-84836351 * Mark bundle as not supporting multiuse < HTTP/1.1  302 Your allowed limit

[gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

2021-05-05 Thread Rahkonen Jukka (MML)
Hi, Have a look at https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m. There gdalinfo fails with /vsicurl/ somewhere in the middle of the grib2 file when the server sends http 302 as a response for a range request. The

Re: [gdal-dev] vsicurl and HTTP error 403

2020-04-09 Thread Ivan Lucena
copy and paste the gdalinfo report. Thanks man, you are the best. Take care, Ivan From: Even Rouault Sent: Wednesday, April 8, 2020 6:41 PM To: gdal-dev@lists.osgeo.org Cc: Ivan Lucena Subject: Re: [gdal-dev] vsicurl and HTTP error 403 Ivan, Experimental

Re: [gdal-dev] vsicurl and HTTP error 403

2020-04-08 Thread Even Rouault
Ivan, Experimentally, I find that this particular server doesn't like HTTP requests without a User-Agent and, more annoying, that it doesn't like Range requests used by /vsicurl/. But as this a gzip file, seeking into it would be a inefficient process. So the following, defining a user agent, a

[gdal-dev] vsicurl and HTTP error 403

2020-04-08 Thread Ivan Lucena
Hi, I am wondering if someone can help to point what I am doing wrong here. This is data that I am trying to access: $ gdalinfo /vsicurl/https://gimms.gsfc.nasa.gov/MODIS/std/GMOD09Q1/tif/NDVI/2019/249/GMOD09Q1.A2019249.08d.latlon.x33y07.6v1.NDVI.tif.gz ERROR 11: HTTP response code: 403 gdalinf

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread aborruso
Hi Even, as usual GDAL and you are really bombastic!! Thank you very much -- Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread Even Rouault
There are 2 issues: - the CSV driver only recognizes files with .csv extension, unless you prefix the filename with CSV: - the underling resource doesn't support HTTP GET requests used by /vsicurl/. Fortunately in that situation, you can use the alternate /vsicurl_streaming/ implementation So a

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread aborruso
Hi Even, Even Rouault-2 wrote > /vsicurl/ should automatically follow links. This is even tested in > https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L230 I have this example CSV input source https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk It replies: HTTP/1.1 301

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread Even Rouault
On mercredi 11 mars 2020 09:53:55 CET aborruso wrote: > Hi Even , > > > Even Rouault-2 wrote > > > No, this is an implementation detail. You'd have to use the curl or other > > HTTP > > API for that. > > today a lot of datasets, especially structured text data, are hosted in > portals that repl

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread aborruso
Hi Even , Even Rouault-2 wrote > No, this is an implementation detail. You'd have to use the curl or other > HTTP > API for that. today a lot of datasets, especially structured text data, are hosted in portals that reply with "HTTP/1.1 301 Moved Permanently". A classic example are the many and

Re: [gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread Even Rouault
On mercredi 11 mars 2020 09:03:05 CET aborruso wrote: > Hi all, > using /vsicurl/ is it possible to native intercept HTTP 3XX response code > and get the final URL of the source? No, this is an implementation detail. You'd have to use the curl or other HTTP API for that. Even -- Spatialys - Ge

[gdal-dev] /vsicurl/ and 3XX response code

2020-03-11 Thread aborruso
Hi all, using /vsicurl/ is it possible to native intercept HTTP 3XX response code and get the final URL of the source? Thank you -- Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.or

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-02 Thread Jeff McKenna
On 2019-06-02 9:35 AM, Andrew C Aitchison wrote: On Sun, 2 Jun 2019, Jeff McKenna wrote: On 2019-06-01 6:52 PM, Joaquim Manuel Freire Lu?s wrote: Is there a way to force reading the grid locally with the HDF5 driver instead of the automatically picked netCDF driver? __

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-02 Thread Joaquim Manuel Freire Luís
Of Andrew C |>Aitchison |>Sent: Sunday, June 2, 2019 2:35 PM |>To: Jeff McKenna |>Cc: gdal-dev@lists.osgeo.org |>Subject: Re: [gdal-dev] vsicurl access to a netcdf file does not see the |>coordinates |> |>On Sun, 2 Jun 2019, Jeff McKenna wrote: |> |>> On 2019-06-

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-02 Thread Andrew C Aitchison
On Sun, 2 Jun 2019, Jeff McKenna wrote: On 2019-06-01 6:52 PM, Joaquim Manuel Freire Lu?s wrote: Is there a way to force reading the grid locally with the HDF5 driver instead of the automatically picked netCDF driver? ___ I come across this need of

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-02 Thread Jeff McKenna
On 2019-06-01 6:52 PM, Joaquim Manuel Freire Luís wrote: |>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It also |>requires the "user-fault file descriptor" Linux specific mechanism. It actually almost works. This command should be equal to just download the file grdc

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-01 Thread Joaquim Manuel Freire Luís
|>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It also |>requires the "user-fault file descriptor" Linux specific mechanism. It actually almost works. This command should be equal to just download the file grdconvert /vsicurl/http://w3.ualg.pt/~jluis/ftp/tmp/idades_v20

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-01 Thread Joaquim Manuel Freire Luís
|>> Rebuilding my HDF+netCDF+GDAL stack, but now when issue the final |>> |>> nmake /f makefile.vc devinstall |> |>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It also |>requires the "user-fault file descriptor" Linux specific mechanism. Ah, I had the impression that re

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-01 Thread Even Rouault
On samedi 1 juin 2019 18:35:15 CEST Joaquim Manuel Freire Luís wrote: > Thanks Even > > Rebuilding my HDF+netCDF+GDAL stack, but now when issue the final > > nmake /f makefile.vc devinstall Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It also requires the "user-fault

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-01 Thread Joaquim Manuel Freire Luís
t;-Original Message- |>From: Even Rouault |>Sent: Saturday, June 1, 2019 9:38 AM |>To: gdal-dev@lists.osgeo.org |>Cc: Joaquim Manuel Freire Luís |>Subject: Re: [gdal-dev] vsicurl access to a netcdf file does not see the |>coordinates |> |>On vendredi 31 mai 2019

Re: [gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-06-01 Thread Even Rouault
On vendredi 31 mai 2019 23:49:09 CEST Joaquim Manuel Freire Luís wrote: > Hi, > > GDAL master here > > When I run gdalinfo on a remote nc file, instead of reporting the > coordinates it prints the array size Joaquim, Note that in the /vsicurl/ case, the file gets opened by the HDF5Image driver

[gdal-dev] vsicurl access to a netcdf file does not see the coordinates

2019-05-31 Thread Joaquim Manuel Freire Luís
Hi, GDAL master here When I run gdalinfo on a remote nc file, instead of reporting the coordinates it prints the array size gdalinfo /vsicurl/http://w3.ualg.pt/~jluis/ftp/tmp/idades_v2012.grd Driver: HDF5Image/HDF5 Dataset Files: none associated Size is 2029, 1168 Metadata: Conventions=COARDS

Re: [gdal-dev] '/vsicurl/https://` does not exist in the file system, and is not recognized as a supported dataset name.

2018-05-10 Thread Even Rouault
Guy, > We are encountering some issue with vsicurl, we are trying to use VSICURL > to do range queries on rasters that are in azure blob storage, > > We convert the azure blob to https, and in most cases it works well for us. > > In our execution log we see the following message: > > > '/vsicu

[gdal-dev] '/vsicurl/https://` does not exist in the file system, and is not recognized as a supported dataset name.

2018-05-10 Thread Guy Doulberg
Hi, We are encountering some issue with vsicurl, we are trying to use VSICURL to do range queries on rasters that are in azure blob storage, We convert the azure blob to https, and in most cases it works well for us. In our execution log we see the following message: '/vsicurl/https://` does n

Re: [gdal-dev] /vsicurl caching behavior

2018-01-05 Thread Even Rouault
> OK, I understand now the VSICachedFile lifecycle and it makes sense. But > then the disk cache should not be used at all for files in > the CPL_VSIL_CURL_NON_CACHED variable. Indeed. The cache disk code hasn't been touched/tested in years, and CPL_VSIL_CURL_NON_CACHED was added afterwards. >

Re: [gdal-dev] /vsicurl caching behavior

2018-01-05 Thread Patrick Valsecchi
Hi Even, Long time no see! On Fri, Jan 5, 2018 at 12:21 PM, Even Rouault wrote: > > 8) In the case discussed in 7), CPL_VSIL_CURL_NON_CACHED will just purge > > > the data from 1 the 3 caches: papsRegions. The vsil_cache and the disk > will > > > still cache the content. > > > > CPL_VSIL_CURL_N

Re: [gdal-dev] /vsicurl caching behavior

2018-01-05 Thread Even Rouault
Hi, > 1) There are a lot of knobs that can be used to tune the thing that are not > documented. For example CPL_VSIL_CURL_USE_CACHE. Is it on purpose? Yes, the disk cache is an experiment that isn't used anywhere (from what I know) and likely not in a finished state as you noticed in your below

[gdal-dev] /vsicurl caching behavior

2018-01-04 Thread Patrick Valsecchi
Hi, I've been looking at the performance of GDAL in the context of MapServer and QGIS accessing GeoTiff on a HTTP server (S3, mostly). Doing that, I had a good look at the cpl_vsil_curl.cpp file and the surroundings and that raised a few questions/concerns: 1) There are a lot of knobs that can b

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-16 Thread Sean Gillies
Done: https://trac.osgeo.org/gdal/ticket/7095. On Fri, Oct 13, 2017 at 4:29 PM, Even Rouault wrote: > Sean, > > > I'd like to point out that very often URLs in a query string do not need > to > > be encoded. Both the Python (see my earlier example) and Node standard > > parsers will handle the s

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-13 Thread Even Rouault
Sean, > I'd like to point out that very often URLs in a query string do not need to > be encoded. Both the Python (see my earlier example) and Node standard > parsers will handle the string > While researching that issue, and from my pat memories, I found that there is a bit of confusion around

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-13 Thread Sean Gillies
Even, Craig, On Thu, Oct 12, 2017 at 4:15 PM, Even Rouault wrote: > Craig, > > > > > True, but that does eliminate ambiguity in the URL, and does so in a > > > well-known way. > I'd like to point out that very often URLs in a query string do not need to be encoded. Both the Python (see my earli

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-12 Thread Even Rouault
Craig, > True, but that does eliminate ambiguity in the URL, and does so in a > well-known way. > > Does the current scheme use any encoding? No > How would you escape text in > option values that might use `=` and `,` etc? Or are there guaranteed to be > no freeform-text options in these paths

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-12 Thread Craig de Stigter
Hi folks We're slightly invested in this because we use VSI paths reasonably heavily, though not so much for cloud services yet. > One downside is that you need to URLEncode the URL, which can make it > painful when composing it at hand. True, but that does eliminate ambiguity in the URL, and

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-12 Thread Even Rouault
Hi Sean, > Is the future of open and creation options? I don't understand your above sentence. > Do you imagine this extended > to, say, block size, compression, number of threads? An RFC that discussed > the scope of this and at what level of abstraction it is implemented at > might be warrante

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-12 Thread Sean Gillies
Hi Even, On Tue, Oct 10, 2017 at 4:02 AM, Even Rouault wrote: > Hi Sean, > > > > > > > > It's written in > > > http://gdal.org/gdal_virtual_file_systems.html#gdal_ > virtual_file_systems_vsi > > > curl > > > > Starting with GDAL 2.3, options can be passed in the filename with the > > > > > > fol

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-10 Thread Even Rouault
Hi Sean, > > It's written in > http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsi > curl > > Starting with GDAL 2.3, options can be passed in the filename with the > > following syntax: /vsicurl/option1=val1[,optionN=valN]*,url=http://... > > I'd like to discuss the de

Re: [gdal-dev] vsicurl configuration design decisions

2017-10-09 Thread Frank Warmerdam
Sean, The obvious answer is that interfaces that are organized around the dataset name do not make it easy to transport other parameters in a way that is specific to one dataset. While I worry a bit about complex dataset syntaxes I do think there is a power to embedding these options in the datas

[gdal-dev] vsicurl configuration design decisions

2017-10-09 Thread Sean Gillies
Hi all, It's written in http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicurl : > Starting with GDAL 2.3, options can be passed in the filename with the following syntax: /vsicurl/option1=val1[,optionN=valN]*,url=http://... I'd like to discuss the design decisions that

Re: [gdal-dev] /vsicurl/ and gz files

2013-10-11 Thread Jose Gomez-Dans
On 11 October 2013 15:57, Kyle Shannon wrote: > > As Even mentioned, the NetCDF driver doesn't use the VSI*L mechanism, > so you can't use /vsi*/ stuff with it unless you use > LD_PRELOAD=vsipreload.so, see above. > OK, sorry, I thought that only applied to the /vsicurl/ method. Thanks for the e

Re: [gdal-dev] /vsicurl/ and gz files

2013-10-11 Thread Kyle Shannon
On Fri, Oct 11, 2013 at 7:13 AM, Jose Gomez-Dans wrote: > Hello everyone, > > On 10 October 2013 19:58, Even Rouault wrote: > >> Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit : >> > Hi, >> > >> > I was wondering if it's possible to use /vsitar/ to have a peek in .gz >> > files, such

Re: [gdal-dev] /vsicurl/ and gz files

2013-10-11 Thread Jose Gomez-Dans
Hello everyone, On 10 October 2013 19:58, Even Rouault wrote: > Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit : > > Hi, > > > > I was wondering if it's possible to use /vsitar/ to have a peek in .gz > > files, such as this one: > > > http://www.globalbedo.org/GlobAlbedo29/tiles/2004

Re: [gdal-dev] /vsicurl/ and gz files

2013-10-10 Thread Even Rouault
Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit : > Hi, > > I was wondering if it's possible to use /vsitar/ to have a peek in .gz > files, such as this one: > http://www.globalbedo.org/GlobAlbedo29/tiles/2004/h17v04/GlobAlbedo.2004001 > .h17v04.nc.gz This is a gzip file, not a .tar.gz

[gdal-dev] /vsicurl/ and gz files

2013-10-10 Thread Jose Gomez-Dans
Hi, I was wondering if it's possible to use /vsitar/ to have a peek in .gz files, such as this one: http://www.globalbedo.org/GlobAlbedo29/tiles/2004/h17v04/GlobAlbedo.2004001.h17v04.nc.gz The file is a NetCDF, with 20 different layers. I was hoping to be able to read the file in using /vsitar/ (

Re: [gdal-dev] vsicurl

2012-03-23 Thread Kyle Shannon
Thanks. kss On Fri, Mar 23, 2012 at 15:06, Frank Warmerdam wrote: > On Fri, Mar 23, 2012 at 1:57 PM, Kyle Shannon wrote: > > Can the VSI*L support be added to drivers that use a third party API > similar > > to the way it is done in the geotiff driver? Would that be a good > reference > > to

Re: [gdal-dev] vsicurl

2012-03-23 Thread Frank Warmerdam
On Fri, Mar 23, 2012 at 1:57 PM, Kyle Shannon wrote: > Can the VSI*L support be added to drivers that use a third party API similar > to the way it is done in the geotiff driver?  Would that be a good reference > to add support to other drivers? Kyle, There are a number of external libraries whe

Re: [gdal-dev] vsicurl

2012-03-23 Thread Kyle Shannon
Etienne, I saw that under the vsizip section, but it wasn't clear to me if the same constraints were on the vsicurl handler. kss On Fri, Mar 23, 2012 at 14:56, Etienne Tourigny wrote: > Kyle, > > the driver(s) probably need to support "large file API" > > From http://trac.osgeo.org/gdal/wiki/Use

Re: [gdal-dev] vsicurl

2012-03-23 Thread Kyle Shannon
Can the VSI*L support be added to drivers that use a third party API similar to the way it is done in the geotiff driver? Would that be a good reference to add support to other drivers? On Fri, Mar 23, 2012 at 14:33, Frank Warmerdam wrote: > Kyle, > > You need to have built GDAL with curl suppo

Re: [gdal-dev] vsicurl

2012-03-23 Thread Etienne Tourigny
Kyle, the driver(s) probably need to support "large file API" >From http://trac.osgeo.org/gdal/wiki/UserDocs/ReadInZip The fact that this new capability is implemented as virtual file systems imply that it will only work for GDAL drivers supporting the "large file API". A non-exhaustive list of s

Re: [gdal-dev] vsicurl

2012-03-23 Thread Frank Warmerdam
Kyle, You need to have built GDAL with curl support and the drivers has do do all file access through VSI*L functions. Best regards, Frank On Fri, Mar 23, 2012 at 1:31 PM, Kyle Shannon wrote: > What are the requirements for using the vsicurl handler for accessing files? > Do drivers have to imp

[gdal-dev] vsicurl

2012-03-23 Thread Kyle Shannon
What are the requirements for using the vsicurl handler for accessing files? Do drivers have to implement anything special do have curl support? ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev