Even, Christopher: I have started building a new overview with the GDAL_CACHEMAX set to 65. Already, I can see the improvement.
It is a bit embarrassing to inquire why GDAL_CACHEMAX did not show such a change using 2048 or a larger number? I am seeing the thrashing that you mentioned @ 65 so I will increase GDAL_CACHEMAX a little bit at a time. Re: Horizontal Differencing ( LZW Compressed across 16-bits ), the results for elevation data are quite noticeable. The frequency at which the high and low-order bytes change is very different. The uncompressed ASTER GDEM V001 ( elevation and metadata ( NUM ) files ) can be fit on a disk with about 150 GB. That's much better than uncompressed ( 1.2 TB ) and LZW 8-bit compression ( 350 GB IIRC ). As many people probably know, a 16-bit raster dataset may increase in size with LZW 8-bit compression. I have often wondered if the LZW horizontal differencing could be extended to 32-bits and what the implications would be. The source code for the TIFF LZW compression options is fairly clean ( dates back to the mid-80s I think ). Could a floating point number be handled and at what cost ( compressed file size vs. uncompressing for real-time / online access )? Merci beaucoup / Thanks, = David On Tue, Feb 9, 2010 at 3:24 PM, Even Rouault <even.roua...@mines-paris.org> wrote: > 2 things : > > 1) Christopher Schmidt is right in suggesting to increase GDAL_CACHEMAX (the > default is 40 MB). In your use case (building external overviews with > compression), the analysis of the algorithm (and a bit of experimentation to > confirm ;-)) shows that that the minimum cache size is : > source_block_width * dst_block_height * overview_factor * sizeof(datatype) = > 119999 * 128 * 2 * 2 = 61, 4 MB > (to be exact, you should also add to that a bit of extra space for the size of > a target block (128 * 128 * 2) to hold in the cache). So specifying --config > GDAL_CACHEMAX 65 should be far sufficient to avoid source band thrashing. > Increasing it more than that wouldn't help. > > More generally, to diagnose performance problems, setting CPL_DEBUG=ON can be > interesting. If you see, a warning like 'GDAL: Potential thrashing on band 1 > of', this is an indication to try increasing GDAL_CACHEMAX > > 2) --config PREDICTOR 2 will be ignored. This is a creation option of the > GTiff driver, but there's no corresponding configuration option in the GTiff > overview building code. This could be easily added if needed. > > Le Tuesday 09 February 2010 23:32:55 David Fogel, vous avez écrit : >> The gdaladdo command is below (for a single overview): >> >> D:\SRTM\Central_North> "c:\Tamas\bin\gdal\apps\gdaladdo.exe" -r >> nearest -ro --config BIGTIFF_OVERVIEW YES --config COMPRESS_OVERVIEW >> LZW --config PREDICTOR 2 Central_North.tif 2 >> >> >> On Tue, Feb 9, 2010 at 2:26 PM, Even Rouault >> >> <even.roua...@mines-paris.org> wrote: >> > And the exact gdaladdo command line you're using ? (must be sure if it's >> > internal or external overviews) >> > >> > Le Tuesday 09 February 2010 23:22:05 David Fogel, vous avez écrit : >> >> Hi Even: >> >> >> >> The GeoTIFF file is described below (output from gdalinfo). The >> >> pre-compiled code is from Tamas' site: >> >> http://vbkto.dyndns.org/sdk/ >> >> I am using the MSVC2008 version ( scroll to the bottom of the page ). >> >> This ought to be v1.7.0 with some or all of the code for HFA files >> >> folded into it. >> >> >> >> The workstation is running Windows 7 64-bit w/ 8 GB of RAM and too >> >> much disk space to know what to do with .. for the moment. >> >> >> >> Thanks. >> >> = David >> >> >> >> ----- % cut here % ----- >> >> Driver: GTiff/GeoTIFF >> >> Files: Central_North.tif >> >> Size is 119999, 74401 >> >> Coordinate System is: >> >> GEOGCS["WGS 84", >> >> DATUM["WGS_1984", >> >> SPHEROID["WGS 84",6378137,298.257223563, >> >> AUTHORITY["EPSG","7030"]], >> >> AUTHORITY["EPSG","6326"]], >> >> PRIMEM["Greenwich",0], >> >> UNIT["degree",0.0174532925199433], >> >> AUTHORITY["EPSG","4326"]] >> >> Origin = (-31.999583000000001,60.000416333276846) >> >> Pixel Size = (0.000833333333333,-0.000833333333333) >> >> Metadata: >> >> AREA_OR_POINT=Area >> >> Image Structure Metadata: >> >> COMPRESSION=LZW >> >> INTERLEAVE=BAND >> >> Corner Coordinates: >> >> Upper Left ( -31.9995830, 60.0004163) ( 31d59'58.50"W, 60d 0'1.50"N) >> >> Lower Left ( -31.9995830, -2.0004170) ( 31d59'58.50"W, 2d 0'1.50"S) >> >> Upper Right ( 67.9995837, 60.0004163) ( 67d59'58.50"E, 60d 0'1.50"N) >> >> Lower Right ( 67.9995837, -2.0004170) ( 67d59'58.50"E, 2d 0'1.50"S) >> >> Center ( 18.0000003, 28.9999997) ( 18d 0'0.00"E, 29d 0'0.00"N) >> >> Band 1 Block=119999x1 Type=Int16, ColorInterp=Gray >> >> NoData Value=-32768 >> >> Metadata: >> >> LAYER_TYPE=athematic >> >> ----- % cut here % ----- >> >> >> >> On Tue, Feb 9, 2010 at 2:09 PM, Even Rouault >> >> >> >> <even.roua...@mines-paris.org> wrote: >> >> > Could you append the output of gdalinfo on the TIF you make overviews >> >> > on and the exact gdaladdo you're using (mainly if you use one >> >> > particular resampling algorithm) ? >> >> > >> >> > Le Tuesday 09 February 2010 22:57:26 David Fogel, vous avez écrit : >> >> >> Hi: >> >> >> >> >> >> In the process of moving data into GeoTIFF, I am re-creating pyramids >> >> >> (OVR files). The good news: It takes fewer than ten minutes to >> >> >> create an LZW Horizontal Differenced compressed 3.5 GB file on a >> >> >> Windows 7 box using GDAL 1.70+ ( via Tamas Szekeres site; 64-bit >> >> >> precompiled ). >> >> >> >> >> >> The OVR files are problematic. Using gdal_translate, I can resize by >> >> >> 50% in less than ten minutes. Using gdaladdo, the first layer ( "2" >> >> >> ) takes hours to complete. >> >> >> >> >> >> This is for SRTM 16-bit data. It appears that something is amiss. >> >> >> >> >> >> Tips and tricks? >> >> >> >> >> >> Thanks! >> >> >> = David >> >> >> _______________________________________________ >> >> >> gdal-dev mailing list >> >> >> gdal-dev@lists.osgeo.org >> >> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
