Hi Even,
Frank pointed me to your blog entry on the GDAL virtual file system at:
http://erouault.blogspot.com/2012/05/new-gdal-virtual-file-system-to-read.html
This is most excellent. I am trying to get this to work using the
/vsicurl/ option to connect to a php page that generates CSV output, But
it is not sucessful :(
I was hoping you could help me sort out how to get this to work.
Running GDAL 1.9.0
woodbri@mappy:/u/software/gdal-1.9.0$ ./config.status -V
config.status
configured by ./configure, generated by GNU Autoconf 2.68,
with options "'--prefix=/usr/local' '--mandir=/usr/local/share/man'
'--includedir=/usr/local/include/gdal' '--with-threads' '--with-hdf5=no'
'--with-grass=no' '--with-ecw=no' '--with-mrsid=no' '--with-jp2mrsid=no'
'--with-libtiff=internal' '--with-geotiff=internal' '--with-jasper'
'--with-netcdf' '--with-xerces' '--with-geos' '--with-sqlite3'
'--with-curl' '--with-pg' '--with-mysql' '--with-perl' '--with-python'
'--with-ecw' '--with-mrsid=/u/software/Geo_DSDK-7.0.0.2167'
'--with-fgdb=/u/software/FileGDB_API' '--with-cfitsio=no'"
I have created a file test.csv and copied the output of the php script
into and created a test.vrt and ogrinfo is happy with it.
I change test.vrt to
<SrcDataSource>/vsicurl/http://example.com/ws/test-pg.php</SrcDataSource>
This is happy:
woodbri@mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt
INFO: Open of `test-pg.vrt'
using driver `VRT' successful.
1: test (Point)
But this is not:
woodbri@mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt test
INFO: Open of `test-pg.vrt'
using driver `VRT' successful.
ERROR 1: Failed to open datasource
`/vsicurl/http://example.com/ws/test-pg.php'.
Layer name: test
Geometry: Point
Feature Count: 0
Layer SRS WKT:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
TOWGS84[0,0,0,0,0,0,0],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9108"]],
AUTHORITY["EPSG","4326"]]
Doing some additional work like find a php script the simulates byte
range downloading. and turning on some debuging, it looks like I have
run into a but in OGR.
So if you notice below, OGR gets the file size and then asks for more
bytes then are in the file. The whole file is sent but OGR get the 200
response code but (I guessing) because the size returned is smaller then
the size requested, it repeats the request over and over.
I'll grab the 1.9.2 tarball and see if I can build that and get better
results. If your php request is doing a sql query (like I'm doing) then
this is going to be problematic if the results exceed a single request,
because you can not be assured that the subsequent request for the next
part of the file will generate the exact same query results. It would be
very helpful if there is a mode or virtual driver where OGR reads the
whole data stream and buffers it into a tmp file or whatever then deals
with it, because there is no way for the server to put the request into
a tempfile and then relate it to subsequent requests from that client
versus other clients and know when to clean things up. It makes more
sense for the client to handle these issues because it knows when it is
done with the resource.
Any thoughts or ideas on all of this?
Thanks,
-Steve
woodbri@mappy:~/work/mongo/ogr$ ogrinfo --debug on -ro test-pg.vrt test
OGR: OGROpen(test-pg.vrt/0x177c3a0) succeeded as VRT.
INFO: Open of `test-pg.vrt'
using driver `VRT' successful.
OGR: GetLayerCount() = 1
VSICURL: GetFileList(/vsicurl/http://imaptools.com:8080/ws)
VSICURL: File[0] = geoip.php, is_dir = 0, size = 0, time = 2013/02/08
15:44:00
VSICURL: File[1] = test-pg.php, is_dir = 0, size = 0, time = 2013/02/09
15:55:00
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: GetFileSize(http://imaptools.com:8080/ws/test-pg.php)=13053
response_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
^C
I logged the php code and here is a partial output:
header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053
header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053
header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
