Jukka,
Running with CPL_CURL_VERBOSE=YES shows the following
> GET
/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2
HTTP/1.1
Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84819968-84836351
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Your allowed limit has been reached. Please go to
https://www.weather.gov/abusive-user-block for more info
* no chunk, no close, no size. Assume close to signal end
<
So it seems that after a number of quickly consecutive GET requests to
the server as here, the server no longer want to see us. The 302 HTTP
error it sends however is not appropriate. 429 "Too many requests" would
be more appropriate
GRIB files with lots of "messages" / bands like that one aren't really
cloud friendly, as you have to skip in many parts of the file to get an
inventory of it.
That said, I see that in that instance, the .grib files are accompanied
by a sidecar file, like
https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2.idx,
which I believe is produced by the wgrib utility. We could conceptually
try to make use of it, however it doesn't contain all metadata that are
typically retrieved by the driver, so for gdalinfo itself that wouldn't
be sufficient, but for just a Open() call that might be enough.
I've noted that https://github.com/OSGeo/gdal/issues/3799
Even
Le 05/05/2021 à 22:04, Rahkonen Jukka (MML) a écrit :
Hi,
Have a look at
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m
<https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m>.
There gdalinfo fails with /vsicurl/ somewhere in the middle of the
grib2 file when the server sends http 302 as a response for a range
request. The file is OK as was tested by downloading it first totally
with curl. Could it be that the http “moved permanently” at that stage
makes /vsicurl/ to fail?
I made also a few tests and it seems that the error may happen in
another http range, but it is always similar:
VSICURL: Got response_code=302
GRIB: ERROR: Ran out of file in Section 7
-Jukka Rahkonen-
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
