Just a thought, but 2.3GB is remarkably like the maximum size of a positive
32bit integer. Is the file being written on Windows or some other 32bit system?
Peter
Even Rouault wrote:
Quoting Rahkonen Jukka <jukka.rahko...@mmmtike.fi>:
Jukka, you can probably speed up considerably the gdalwarp
process by adding "-co TILED=YES" to the gdalwarp
commandline. For such a big output file, using the default
stripped structure of a TIFF will be very inefficient because
the way gdalwarp proceeds (by sort of tiles) causes a lot of
cache thrashing and I/O. A little of tuning with the "-wm
XXX", "-config GDAL_CACHEMAX YYY" and -multi options might also help.
I can have a try. I want to change just on option at a time and now it
is running with -co TILED=YES option. I will see tomorrow how did it do.
Now I can say just that in the beginning things looked much better.
Memory consumption and processor load were much higher and disk unit
went more silently. But after some half an hour the conversion seems to
have almost stopped at 2,302,729,094 bytes output file size. Gdalwarp
reserves 390 megabytes of memory and utilises 5-15 percent of processor
time.
I have no explanation in mind why it seems to have stalled at 2.3 GB. I would
let it go and see what happens a few hours after.
Can you suggest any reasonable values for other parameters? This old
machine is running on Windows XP Pro and it has only 1 GB memory and one
3 GHz processor.
The -wm defaults to 64 MB and GDAL_CACHEMAX to 40 MB. So increasing both to 300
for example might work (it will let 1000-2*300=400 MB for the rest of the OS and
other buffers of gdalwarp). If you see the computer starts swapping, those
values should be reduced. See http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp
for a discussion on those parameters.
I do not have myself a real need for making large mosaics through .vrt
files at the moment, but as the set-up is ready and the computer
otherwise idle I can well let it run and gather some hopefully
comparable numbers. I have also a bit more powerful computer with 8
cores that I can use for final run if there is a reason to believe that
gdalwarp can do the job with some settings. I guess -multi stands for
multithreading and it may suit well the 8-core system.
-multi can also be helpfull with a single core system. It uses a thread for I/O
and another one for computation. The I/O thread is mostly I/O bound and doesn't
generally consumes much CPU, except if you use a compressed input or output file
(which might be the case with your JP2K data). But having more than 2 cores
won't help much.
-Jukka-
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
--
--------------------------------------------------------------------------------
Peter J Halls, GIS Advisor, University of York
Telephone: 01904 433806 Fax: 01904 433740
Snail mail: Computing Service, University of York, Heslington, York YO10 5DD
This message has the status of a private and personal communication
--------------------------------------------------------------------------------
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
