Personally I would agree with your approach at looking at the cost of running the formula first.
However if you do change your code to read/write blocks than there are additional opportunities for speeding the processing up. Once you have the block size reading working consider using multiple threads. You will need to use a GDALDataset for each thread. You will however need to consider how you write the results out again as you will not be able to write to a single output. You will have to write to a separate raster for each thread. This is the approach I have taken. I can't help you with writing the block back as once I've got the raster data I processes it into a format that allows me to load and display the data a lot quicker than going back to GDAL for the data. You need to find the equivalent to: http://www.gdal.org/classGDALRasterBand.html#a30786c81246455321e96d73047b8edf1 Also these may help... http://www.gdal.org/classGDALRasterBand.html#ad80cecc562fd48f5783677de625360ac http://www.gdal.org/classGDALRasterBand.html#a457ae6645dd2680289734aed0afce357 On 15 June 2017 at 14:04, Paul Meems <bontepaar...@gmail.com> wrote: > Thanks all for your suggestions. > > @Rutger and @Damian: > Thanks for your suggestion about the blocks. > I had a look at Rutger's links, > I create the input file myself so I can add 'TILES=YES' but I'm not sure > how to change my calculation code. > I see in this first link xbs, ybs = b1.GetBlockSize() But I don't see > when to use the xbs or ybs variables. > I assume I need to change the reading of the data: band.ReadRaster(0, > line, sizeX, 1, scanline, sizeX, 1, 0, 0); but I'm not sure how. > And how should I write back the block? > > BTW. The main bottle-neck seems to be in the used formula parser. > With the formula it takes 68s, without the formule - just setting the > pixelvalue takes 3.3s and with the formula written in code pixelValue = > (float)Math.Exp(3.1 + 0.9 * f); it only takes 3.4s. > So not using mathparser has the highest benefits. I will do that first, > but I also want to understand the block reading and writing. > > @Jan, thanks for your link to the other parser. I had a quick look and it > looks very promising. Sadly I couldn't get their C# example working. > I will look at as well. > > Just a more general question. > Doesn't it makes sense if gdal would provide a gdal_calculate tool which > also is librified like VectorTranslate? > It seems lots of people are implementing it on their own. > > Thanks, > > Paul > > > 2017-06-14 9:29 GMT+02:00 Rutger <kass...@gmail.com>: > >> Damian Dixon wrote >> > It is usually better to process the pixels in a block rather than >> walking >> > across each row especially if you are processing a TIFF as these are >> > usually stored as tiles (blocks). >> >> Other layouts are common as well. For example, the Landsat TIFF's provided >> by the USGS have a row based-layout. If you can choose it yourself, bocks >> are prefered in my opinion, since GDAL VRT's have a fixed blocksize >> of128*128. So when writing TIFF's, setting "TILED=YES" is a good default. >> >> I think your spot on by mentioning the blocks. Don't assume the layout at >> all, look at the blocksize of the file and use it. If the blocks are >> relatively small (memoy-wise), using a multiple of the size can increase >> performance a bit more. So if its row-based and you have plenty of memory >> to >> spare, why not read blocks of (xsize, 128). Or if the blocksize is >> 128x128, >> use blocks of 256x256 etc. >> >> If the volume of data is really large, increasing GDAL's block cache can >> be >> helpful. Although its best to avoid relying on the cache (if possible) by >> specifying an appropriate blocksize. >> >> Here are a few mini-benchmarks: >> http://nbviewer.jupyter.org/gist/RutgerK/4faa43873ee6389873dd4de85de7c450 >> >> https://stackoverflow.com/questions/41742162/gdal-readasarra >> y-for-vrt-extremely-slow/41853759#41853759 >> >> Regards, >> Rutger >> >> >> >> >> -- >> View this message in context: http://osgeo-org.1560.x6.nabbl >> e.com/gdal-dev-Raster-calculation-optimalisation-tp5324014p5324120.html >> Sent from the GDAL - Dev mailing list archive at Nabble.com. >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> > > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
