Fantastic Even. Thanks a lot.
(and ofc I meant to say grib2, not “gring2”)
Joaquim
From: Even Rouault
Sent: Thursday, May 8, 2025 11:04 PM
To: Joaquim Manuel Freire Luís ; gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] vsicurl and --range byte-range
Joaquim,
You can combine /vsisubfile
Joaquim,
You can combine /vsisubfile/
(https://gdal.org/en/stable/user/virtual_file_systems.html#vsisubfile-portions-of-files)
and /vsicurl/
/vsisubfile/${offset}_${length}/vsicurl/${url}
e.g.
$ gdalinfo
/vsisubfile/802533_508958,/vsicurl/https://ecmwf-forecasts.s3.eu-central-1.amazonaws.c
Hi,
I'm writing a tool to download ECMWF forecast data in gring2 format. But since
those files have > 150 bands and most of times I'm only interested in a couple
of them, I'm using the only documented mechanism that I found [1], which is the
"curl -range start-stop". That works fine but what I
On Thu, Jan 5, 2023 at 7:14 PM Joe Lee wrote:
>
> Hi, Paolo!
>
>
>
> I tested on mac:
>
>
>
> Darwin nene.ad.hdfgroup.org 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6
> 23:31:16 PST 2022; root:xnu-8020.240.14~1/RELEASE_X86_64 x86_64
>
>
>
> I’m using `brew install gdal.`
>
>
>
> I just tested
-dev on behalf of Paolo Corti
Date: Thursday, January 5, 2023 at 12:10 PM
To: Even Rouault
Cc: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] vsicurl and HDF5
On Thu, Jan 5, 2023 at 6:03 PM Even Rouault wrote:
>
> Paolo,
>
> Le 05/01/2023 à 17:56, Paolo Corti a écrit :
> >
On Thu, Jan 5, 2023 at 6:03 PM Even Rouault wrote:
>
> Paolo,
>
> Le 05/01/2023 à 17:56, Paolo Corti a écrit :
> > Hi all
> >
> > Do you know if I can read HDF5 files with vsicurl?
> > I made a few tests, but I am getting this error, so I was wondering it
> > it is even possible:
>
> yes this is p
ill=No
HDFEOS_GRIDS_VNP_Grid_DNB_Data_Fields_BrightnessTemperature_M12_add_offset=203
…
From: gdal-dev on behalf of Paolo Corti
Date: Thursday, January 5, 2023 at 10:57 AM
To: gdal-dev@lists.osgeo.org
Subject: [gdal-dev] vsicurl and HDF5
Hi all
Do you know if I can read HDF5 files with vsicurl?
I made a few tests, but I am getting this error, so
Paolo,
Le 05/01/2023 à 17:56, Paolo Corti a écrit :
Hi all
Do you know if I can read HDF5 files with vsicurl?
I made a few tests, but I am getting this error, so I was wondering it
it is even possible:
yes this is possible. If you get the error message you mention, it means
that the HDF5 dri
Hi all
Do you know if I can read HDF5 files with vsicurl?
I made a few tests, but I am getting this error, so I was wondering it
it is even possible:
gdalinfo /vsicurl/https://./myfile.h5
ERROR 1: CPLCreateUserFaultMapping(): syscall(__NR_userfaultfd)
failed: error = 38
gdalinfo failed - un
Jukka,
Running with CPL_CURL_VERBOSE=YES shows the following
> GET
/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2
HTTP/1.1
Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84819968-84836351
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Your allowed limit
Hi,
Have a look at
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m.
There gdalinfo fails with /vsicurl/ somewhere in the middle of the grib2 file
when the server sends http 302 as a response for a range request. The
copy and paste the
gdalinfo report.
Thanks man, you are the best. Take care,
Ivan
From: Even Rouault
Sent: Wednesday, April 8, 2020 6:41 PM
To: gdal-dev@lists.osgeo.org
Cc: Ivan Lucena
Subject: Re: [gdal-dev] vsicurl and HTTP error 403
Ivan,
Experimental
Ivan,
Experimentally, I find that this particular server doesn't like HTTP requests
without a
User-Agent and, more annoying, that it doesn't like Range requests used by
/vsicurl/.
But as this a gzip file, seeking into it would be a inefficient process.
So the following, defining a user agent, a
Hi,
I am wondering if someone can help to point what I am doing wrong here.
This is data that I am trying to access:
$ gdalinfo
/vsicurl/https://gimms.gsfc.nasa.gov/MODIS/std/GMOD09Q1/tif/NDVI/2019/249/GMOD09Q1.A2019249.08d.latlon.x33y07.6v1.NDVI.tif.gz
ERROR 11: HTTP response code: 403
gdalinf
Hi Even,
as usual GDAL and you are really bombastic!!
Thank you very much
--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
There are 2 issues:
- the CSV driver only recognizes files with .csv extension, unless you prefix
the filename with CSV:
- the underling resource doesn't support HTTP GET requests used by /vsicurl/.
Fortunately in that situation, you can use the alternate /vsicurl_streaming/
implementation
So a
Hi Even,
Even Rouault-2 wrote
> /vsicurl/ should automatically follow links. This is even tested in
> https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L230
I have this example CSV input source
https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk
It replies:
HTTP/1.1 301
On mercredi 11 mars 2020 09:53:55 CET aborruso wrote:
> Hi Even ,
>
>
> Even Rouault-2 wrote
>
> > No, this is an implementation detail. You'd have to use the curl or other
> > HTTP
> > API for that.
>
> today a lot of datasets, especially structured text data, are hosted in
> portals that repl
Hi Even ,
Even Rouault-2 wrote
> No, this is an implementation detail. You'd have to use the curl or other
> HTTP
> API for that.
today a lot of datasets, especially structured text data, are hosted in
portals that reply with "HTTP/1.1 301 Moved Permanently".
A classic example are the many and
On mercredi 11 mars 2020 09:03:05 CET aborruso wrote:
> Hi all,
> using /vsicurl/ is it possible to native intercept HTTP 3XX response code
> and get the final URL of the source?
No, this is an implementation detail. You'd have to use the curl or other HTTP
API for that.
Even
--
Spatialys - Ge
Hi all,
using /vsicurl/ is it possible to native intercept HTTP 3XX response code
and get the final URL of the source?
Thank you
--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.or
On 2019-06-02 9:35 AM, Andrew C Aitchison wrote:
On Sun, 2 Jun 2019, Jeff McKenna wrote:
On 2019-06-01 6:52 PM, Joaquim Manuel Freire Lu?s wrote:
Is there a way to force reading the grid locally with the HDF5 driver
instead of the automatically picked netCDF driver?
__
Of Andrew C
|>Aitchison
|>Sent: Sunday, June 2, 2019 2:35 PM
|>To: Jeff McKenna
|>Cc: gdal-dev@lists.osgeo.org
|>Subject: Re: [gdal-dev] vsicurl access to a netcdf file does not see the
|>coordinates
|>
|>On Sun, 2 Jun 2019, Jeff McKenna wrote:
|>
|>> On 2019-06-
On Sun, 2 Jun 2019, Jeff McKenna wrote:
On 2019-06-01 6:52 PM, Joaquim Manuel Freire Lu?s wrote:
Is there a way to force reading the grid locally with the HDF5 driver
instead of the automatically picked netCDF driver?
___
I come across this need of
On 2019-06-01 6:52 PM, Joaquim Manuel Freire Luís wrote:
|>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It
also
|>requires the "user-fault file descriptor" Linux specific mechanism.
It actually almost works. This command should be equal to just download the file
grdc
|>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It
also
|>requires the "user-fault file descriptor" Linux specific mechanism.
It actually almost works. This command should be equal to just download the file
grdconvert /vsicurl/http://w3.ualg.pt/~jluis/ftp/tmp/idades_v20
|>> Rebuilding my HDF+netCDF+GDAL stack, but now when issue the final
|>>
|>> nmake /f makefile.vc devinstall
|>
|>Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It
also
|>requires the "user-fault file descriptor" Linux specific mechanism.
Ah, I had the impression that re
On samedi 1 juin 2019 18:35:15 CEST Joaquim Manuel Freire Luís wrote:
> Thanks Even
>
> Rebuilding my HDF+netCDF+GDAL stack, but now when issue the final
>
> nmake /f makefile.vc devinstall
Ah, there is no chance that /vsicurl/ with netcdf will work on Windows. It
also requires the "user-fault
t;-Original Message-
|>From: Even Rouault
|>Sent: Saturday, June 1, 2019 9:38 AM
|>To: gdal-dev@lists.osgeo.org
|>Cc: Joaquim Manuel Freire Luís
|>Subject: Re: [gdal-dev] vsicurl access to a netcdf file does not see the
|>coordinates
|>
|>On vendredi 31 mai 2019
On vendredi 31 mai 2019 23:49:09 CEST Joaquim Manuel Freire Luís wrote:
> Hi,
>
> GDAL master here
>
> When I run gdalinfo on a remote nc file, instead of reporting the
> coordinates it prints the array size
Joaquim,
Note that in the /vsicurl/ case, the file gets opened by the HDF5Image driver
Hi,
GDAL master here
When I run gdalinfo on a remote nc file, instead of reporting the coordinates
it prints the array size
gdalinfo /vsicurl/http://w3.ualg.pt/~jluis/ftp/tmp/idades_v2012.grd
Driver: HDF5Image/HDF5 Dataset
Files: none associated
Size is 2029, 1168
Metadata:
Conventions=COARDS
Guy,
> We are encountering some issue with vsicurl, we are trying to use VSICURL
> to do range queries on rasters that are in azure blob storage,
>
> We convert the azure blob to https, and in most cases it works well for us.
>
> In our execution log we see the following message:
>
>
> '/vsicu
Hi,
We are encountering some issue with vsicurl, we are trying to use VSICURL
to do range queries on rasters that are in azure blob storage,
We convert the azure blob to https, and in most cases it works well for us.
In our execution log we see the following message:
'/vsicurl/https://` does n
> OK, I understand now the VSICachedFile lifecycle and it makes sense. But
> then the disk cache should not be used at all for files in
> the CPL_VSIL_CURL_NON_CACHED variable.
Indeed. The cache disk code hasn't been touched/tested in years, and
CPL_VSIL_CURL_NON_CACHED was added afterwards.
>
Hi Even,
Long time no see!
On Fri, Jan 5, 2018 at 12:21 PM, Even Rouault
wrote:
> > 8) In the case discussed in 7), CPL_VSIL_CURL_NON_CACHED will just purge
>
> > the data from 1 the 3 caches: papsRegions. The vsil_cache and the disk
> will
>
> > still cache the content.
>
>
>
> CPL_VSIL_CURL_N
Hi,
> 1) There are a lot of knobs that can be used to tune the thing that are not
> documented. For example CPL_VSIL_CURL_USE_CACHE. Is it on purpose?
Yes, the disk cache is an experiment that isn't used anywhere (from what I
know) and likely
not in a finished state as you noticed in your below
Hi,
I've been looking at the performance of GDAL in the context of MapServer
and QGIS accessing GeoTiff on a HTTP server (S3, mostly).
Doing that, I had a good look at the cpl_vsil_curl.cpp file and the
surroundings and that raised a few questions/concerns:
1) There are a lot of knobs that can b
Done: https://trac.osgeo.org/gdal/ticket/7095.
On Fri, Oct 13, 2017 at 4:29 PM, Even Rouault
wrote:
> Sean,
>
> > I'd like to point out that very often URLs in a query string do not need
> to
> > be encoded. Both the Python (see my earlier example) and Node standard
> > parsers will handle the s
Sean,
> I'd like to point out that very often URLs in a query string do not need to
> be encoded. Both the Python (see my earlier example) and Node standard
> parsers will handle the string
>
While researching that issue, and from my pat memories, I found that there is
a bit of confusion around
Even, Craig,
On Thu, Oct 12, 2017 at 4:15 PM, Even Rouault
wrote:
> Craig,
>
>
>
> > True, but that does eliminate ambiguity in the URL, and does so in a
>
> > well-known way.
>
I'd like to point out that very often URLs in a query string do not need to
be encoded. Both the Python (see my earli
Craig,
> True, but that does eliminate ambiguity in the URL, and does so in a
> well-known way.
>
> Does the current scheme use any encoding?
No
> How would you escape text in
> option values that might use `=` and `,` etc? Or are there guaranteed to be
> no freeform-text options in these paths
Hi folks
We're slightly invested in this because we use VSI paths reasonably
heavily, though not so much for cloud services yet.
> One downside is that you need to URLEncode the URL, which can make it
> painful when composing it at hand.
True, but that does eliminate ambiguity in the URL, and
Hi Sean,
> Is the future of open and creation options?
I don't understand your above sentence.
> Do you imagine this extended
> to, say, block size, compression, number of threads? An RFC that discussed
> the scope of this and at what level of abstraction it is implemented at
> might be warrante
Hi Even,
On Tue, Oct 10, 2017 at 4:02 AM, Even Rouault
wrote:
> Hi Sean,
>
>
>
> >
>
> > It's written in
>
> > http://gdal.org/gdal_virtual_file_systems.html#gdal_
> virtual_file_systems_vsi
>
> > curl
>
> > > Starting with GDAL 2.3, options can be passed in the filename with the
>
> >
>
> > fol
Hi Sean,
>
> It's written in
> http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsi
> curl
> > Starting with GDAL 2.3, options can be passed in the filename with the
>
> following syntax: /vsicurl/option1=val1[,optionN=valN]*,url=http://...
>
> I'd like to discuss the de
Sean,
The obvious answer is that interfaces that are organized around the
dataset name do not make it easy to transport other parameters in a
way that is specific to one dataset.
While I worry a bit about complex dataset syntaxes I do think there is
a power to embedding these options in the datas
Hi all,
It's written in
http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicurl
:
> Starting with GDAL 2.3, options can be passed in the filename with the
following syntax: /vsicurl/option1=val1[,optionN=valN]*,url=http://...
I'd like to discuss the design decisions that
On 11 October 2013 15:57, Kyle Shannon wrote:
>
> As Even mentioned, the NetCDF driver doesn't use the VSI*L mechanism,
> so you can't use /vsi*/ stuff with it unless you use
> LD_PRELOAD=vsipreload.so, see above.
>
OK, sorry, I thought that only applied to the /vsicurl/ method. Thanks for
the e
On Fri, Oct 11, 2013 at 7:13 AM, Jose Gomez-Dans wrote:
> Hello everyone,
>
> On 10 October 2013 19:58, Even Rouault wrote:
>
>> Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit :
>> > Hi,
>> >
>> > I was wondering if it's possible to use /vsitar/ to have a peek in .gz
>> > files, such
Hello everyone,
On 10 October 2013 19:58, Even Rouault wrote:
> Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit :
> > Hi,
> >
> > I was wondering if it's possible to use /vsitar/ to have a peek in .gz
> > files, such as this one:
> >
> http://www.globalbedo.org/GlobAlbedo29/tiles/2004
Le jeudi 10 octobre 2013 20:33:49, Jose Gomez-Dans a écrit :
> Hi,
>
> I was wondering if it's possible to use /vsitar/ to have a peek in .gz
> files, such as this one:
> http://www.globalbedo.org/GlobAlbedo29/tiles/2004/h17v04/GlobAlbedo.2004001
> .h17v04.nc.gz
This is a gzip file, not a .tar.gz
Hi,
I was wondering if it's possible to use /vsitar/ to have a peek in .gz
files, such as this one:
http://www.globalbedo.org/GlobAlbedo29/tiles/2004/h17v04/GlobAlbedo.2004001.h17v04.nc.gz
The file is a NetCDF, with 20 different layers. I was hoping to be able to
read the file in using /vsitar/ (
Thanks.
kss
On Fri, Mar 23, 2012 at 15:06, Frank Warmerdam wrote:
> On Fri, Mar 23, 2012 at 1:57 PM, Kyle Shannon wrote:
> > Can the VSI*L support be added to drivers that use a third party API
> similar
> > to the way it is done in the geotiff driver? Would that be a good
> reference
> > to
On Fri, Mar 23, 2012 at 1:57 PM, Kyle Shannon wrote:
> Can the VSI*L support be added to drivers that use a third party API similar
> to the way it is done in the geotiff driver? Would that be a good reference
> to add support to other drivers?
Kyle,
There are a number of external libraries whe
Etienne,
I saw that under the vsizip section, but it wasn't clear to me if the same
constraints were on the vsicurl handler.
kss
On Fri, Mar 23, 2012 at 14:56, Etienne Tourigny wrote:
> Kyle,
>
> the driver(s) probably need to support "large file API"
>
> From http://trac.osgeo.org/gdal/wiki/Use
Can the VSI*L support be added to drivers that use a third party API
similar to the way it is done in the geotiff driver? Would that be a good
reference to add support to other drivers?
On Fri, Mar 23, 2012 at 14:33, Frank Warmerdam wrote:
> Kyle,
>
> You need to have built GDAL with curl suppo
Kyle,
the driver(s) probably need to support "large file API"
>From http://trac.osgeo.org/gdal/wiki/UserDocs/ReadInZip
The fact that this new capability is implemented as virtual file
systems imply that it will only work for GDAL drivers supporting the
"large file API". A non-exhaustive list of s
Kyle,
You need to have built GDAL with curl support and the drivers has do
do all file access through VSI*L functions.
Best regards,
Frank
On Fri, Mar 23, 2012 at 1:31 PM, Kyle Shannon wrote:
> What are the requirements for using the vsicurl handler for accessing files?
> Do drivers have to imp
What are the requirements for using the vsicurl handler for accessing
files? Do drivers have to implement anything special do have curl support?
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
59 matches
Mail list logo
