Hi Aaron & Even,
Decoding at preccints level in OpenJPEG would definitely be a great
addition.
I believe OpenJPEG performance could also be improved if multi-threading
was managed directly inside the lib, since in the current situation
OpenJPEG is purely monothreaded.
From my experience, Kakadu does a great job in keeping all your cores
occupied close to 100% during decoding, even when decoding a single tile
(so this is probably linked to the capability to manage independent
decoding at the precinct level).
In 3 geospatial software I know leverage on OpenJPEG (gdal, orfeo
toolbox, and snap) to decode big datasets (think tens or hundreds of
thousand pixels in each dimension), multithreading is handled from the
user side, by creating a number of independent contexts and run them in
separate threads to improve performance.
This means we are all interpreting the codestreams and seeking into them
independently, which clearly sounds suboptimal.
I suspect managing multi-threading at a lower level directly inside
OpenJPEG could open the doors to much better resource management (cores
occupation, reduced I/O, etc...).
I certainly concur with Even with the "as fast as it can be" :)
Any percent you gain has a direct visible impact when (de)coding huge
datasets.
Regards,
Julien
On 20/12/2015 11:00, Even Rouault wrote:
Aaron,
I am working on some performance enhancements for OpenJPEG,
and I am curious about people's experience using the OpenJPEG driver with
GDAL:
Does it have all the features you need, if not what is missing?
Being the developer of the driver, I believe it must be close to using the
available capabilities of OpenJPEG as best as possible. I may have overlooked
some features, so other eyes checking the code are always welcome.
From my point of view what is mainly missing in OpenJPEG (the lib):
- ability to decode efficiently sub-windows inside a tile (I guess this means
using precincts). This is particuarly true for single-tiled big files
- reduce memory consumption. I've found that to decode a 2048x2048 tile of a
Sentinel2 product, it takes 600 MB of memory whereas the uncompressed size of
the tile is 8 MB (2048x2048*sizeof(int16)).
- ability to open files who have more than 2 gigapixels (or 4, I'm not sure
where the limit is)
- currently the driver closes and reopen the file each time it reads a tile,
because there are(were?) some instabilities in seeking through arbitrary tiles
(ie reading potentially not in ascending tile order). Would be nice to be able
to avoid that
Given that it is the slowest J2K library around, what would be acceptable
performance in the geo-spatial world?
As fast as possible :-)
These changes apply to the current master version of OpenJPEG, which will
be released shortly.
Were you refering to this PR : https://github.com/uclouvain/openjpeg/pull/668
?
Even
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
