Quoting Jukka Rahkonen <jukka.rahko...@mmmtike.fi>: > Hermann Peifer <peifer <at> gmx.eu> writes: > > > > > Jukka wrote > > > > > > > Did you try to use gdalwarp also for the second step with -srcnodata > value > [value...]? I believe you did set > > some NODATA value in step 1 for the intermediate images. > > > > > > -Jukka Rahkonen- > > > > I did. However, the main problem with gdalwarp is its performance. So my > options seem to be: > > > > 1. > > gdalwarping 1500 ASTER tiles in 1 run into a 4GB 100m raster output.tif > > = 5-10 days processing (my best estimate) > > > > 2. > > gdalwarping 1500 ASTER tiles 1-by-1 as an intermediate step, then > > gdal_merge or gdal_translate to mosaic warped tiles into output.tif > > = smaller or larger NODATA areas in output.tif, along tile borders > > Hi, > > I decided to do a little test in order to see how slow gdalwarp in in this > kind > of usage with my own imagery. My test plan was like this: > > 1. Original images: 199 JPEG2000 images, with 3 bands and 10000 by 10000 > pixels > each. All together make 28 gigabytes on disk in lossless JPEG2000 format, or > 60 > GB uncompressed. > > 2. My first try with gdalbuildvtr > gdalbuildvrt -input_file_list gdalwarptest.txt gdalwarptest.vrt > 0...10...20...30...40...50...60...70...80...90...100 - done. > Nice experience, I like this little tool. > > 3. A try to make a tiff file from this virtual raster with gdalwarp > gdalwarp -co BIGTIFF=YES -srcnodata 0 gdalwarptest.vrt gdalwarptest.tif > Creating output file that is 182026P x 191204L. > > > Gdalwarp has now been running for 27 hours. Process is taking 70 megabytes of > memory and 5-15 percent of processor time. As a result I have now output tiff > file that is very, very slowly growing in size. Now, after 27-hour run the > filesize is 13 gigabytes. Judged by the filesize my new tiff file might be > ready > after 4-5 more days. However, I am not sure if this estimate is reliable. > The > gdalwarp status indicator is still showing just zero for me: > Processing input file gdalwarptest.vrt. > 0 > > Conclusion: Too slow to be useful with my settings. > Question 1: Should it work this way? > Question 2: If yes, is it worth trying to find the limits when it goes too > slow > and perhaps file a ticket?
Jukka, you can probably speed up considerably the gdalwarp process by adding "-co TILED=YES" to the gdalwarp commandline. For such a big output file, using the default stripped structure of a TIFF will be very inefficient because the way gdalwarp proceeds (by sort of tiles) causes a lot of cache thrashing and I/O. A little of tuning with the "-wm XXX", "-config GDAL_CACHEMAX YYY" and -multi options might also help. > > -Jukka Rahkonen- > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
