Hi, Yes, we are pretty sure we will see a significant benefit. The processing algorithms are CPU bound not io bound. Our digital terrain model interpolations often run for many hours ( we do them overnight) but the underlying file is only a few gigabytes. If we split them into multiple files of tiles and run each on a dedicated process the whole thing is quicker, but this is messy and results in a stitching error.
Another example is gdalwarp. It takes quite some time with a large data set and would be. A good candidate for parallelisation, as would gdaladdo. I believe slower cores but more of them in pcs are the future. My pc has 8 but they rarely get used to their potential. I am certain there are some challenges here, that's why it is interesting;) Regards pk On 11/01/2013, at 6:54 PM, "Even Rouault" <even.roua...@mines-paris.org> wrote: > Hi, > > This is an intersting topic, with many "intersecting" issues to deal with at > different levels. > > First, are you confident that in the use cases you imagine that I/O access > won't > be the limiting factor, in which case serialization of I/O could be acceptable > and this would just require an API with a dataset level mutex. > > There are several places where parallel write should be addressed : > - The GDAL core mechanisms that deal with the block cache > - Each GDAL driver where parallel write would be supported. I guess that GDAL > drivers should advertize a specific capability > - The low-level library used by the driver. In the case of GDAL, libtiff > > And finally, as Frank underlined, there are intrinsic limitations due to the > format itself. For a compressed TIFF, at some point, you have to serialize the > writing of the tile, because you cannot kown in advance the size of the > compressed data, or at least have some coordination of the writers so that a > "next offset available" is properly synchronized between them. The compression > itself could be serialized. > > I'm not sure however if what Jan mentionned, different process, writing the > same > dataset is doable. >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
