---------- Forwarded message ---------- From: Aaron Boxer <boxe...@gmail.com> Date: Mon, Dec 21, 2015 at 10:27 AM Subject: Re: [gdal-dev] OpenJPEG To: Julien Malik <julien.ma...@c-s.fr>
Hi Julien, > Decoding at preccints level in OpenJPEG would definitely be a great > addition. > Yes, sounds useful. > > I believe OpenJPEG performance could also be improved if multi-threading > was managed directly inside the lib, since in the current situation > OpenJPEG is purely monothreaded. > From my experience, Kakadu does a great job in keeping all your cores > occupied close to 100% during decoding, even when decoding a single tile > (so this is probably linked to the capability to manage independent > decoding at the precinct level). > In 3 geospatial software I know leverage on OpenJPEG (gdal, orfeo toolbox, > and snap) to decode big datasets (think tens or hundreds of thousand pixels > in each dimension), multithreading is handled from the user side, by > creating a number of independent contexts and run them in separate threads > to improve performance. > This means we are all interpreting the codestreams and seeking into them > independently, which clearly sounds suboptimal. > I suspect managing multi-threading at a lower level directly inside > OpenJPEG could open the doors to much better resource management (cores > occupation, reduced I/O, etc...). > The pull request I mentioned does use all cores for the computationally intensive part of decoding. Reading through the file is still single threaded. And this can take a good chunk of time. > I certainly concur with Even with the "as fast as it can be" :) > Any percent you gain has a direct visible impact when (de)coding huge > datasets. > > Thanks for the feedback. Good news for OpenJPEG performance : they will be *hiring* a developer next year to focus solely on performance. If you guys are interested, please check out the OpenJPEG mailing list and you can apply. Also, I would encourage submitting all of these great ideas as issues on the openjpeg github site: https://github.com/uclouvain/openjpeg/issues Cheers, Aaron On 20/12/2015 11:00, Even Rouault wrote: > Aaron, > > I am working on some performance enhancements for OpenJPEG, >> and I am curious about people's experience using the OpenJPEG driver with >> GDAL: >> >> Does it have all the features you need, if not what is missing? >> > Being the developer of the driver, I believe it must be close to using the > available capabilities of OpenJPEG as best as possible. I may have > overlooked > some features, so other eyes checking the code are always welcome. > > From my point of view what is mainly missing in OpenJPEG (the lib): > - ability to decode efficiently sub-windows inside a tile (I guess this > means > using precincts). This is particuarly true for single-tiled big files > - reduce memory consumption. I've found that to decode a 2048x2048 tile of > a > Sentinel2 product, it takes 600 MB of memory whereas the uncompressed size > of > the tile is 8 MB (2048x2048*sizeof(int16)). > - ability to open files who have more than 2 gigapixels (or 4, I'm not sure > where the limit is) > - currently the driver closes and reopen the file each time it reads a > tile, > because there are(were?) some instabilities in seeking through arbitrary > tiles > (ie reading potentially not in ascending tile order). Would be nice to be > able > to avoid that > > Given that it is the slowest J2K library around, what would be acceptable >> performance in the geo-spatial world? >> > As fast as possible :-) > > These changes apply to the current master version of OpenJPEG, which will >> be released shortly. >> > Were you refering to this PR : > https://github.com/uclouvain/openjpeg/pull/668 > ? > > Even > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
