Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-10 Thread Howard Butler via gdal-dev
> On Dec 8, 2023, at 1:04 PM, Laurențiu Nicola via gdal-dev > wrote: > > - GeoTIFF DEFLATE 280 MB > - Zarr BLOSC zlib NONE 281 MB > - Zarr BLOSC zlib BIT 253 MB > - Zarr BLOSC zlib BYTE 249 MB If floating point compression performance in TIFFs is the primary concern, don't sleep on inves

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Laurențiu Nicola via gdal-dev
On Fri, Dec 8, 2023, at 21:32, Even Rouault wrote: > yes, poor wording of mine. I meant that if using PREDICTOR=3, one should > compare with FILTER=DELTA. But looking more closely, they are not strictly > equivalent. PREDICTOR=3 applies the delta as b[0]-a[0], b[1]-a[1], b[2]-a[2], > b[3]-a[3] w

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Even Rouault via gdal-dev
Le 08/12/2023 à 20:23, Laurențiu Nicola via gdal-dev a écrit : On Fri, Dec 8, 2023, at 21:11, Even Rouault wrote:  - GeoTIFF DEFLATE 280 MB Is it with PREDICTOR=3 ?  (If not, then the ZARR datasets should also be compressed with FILTER=DELTA to have a fair comparison) I would expect a fa

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Laurențiu Nicola via gdal-dev
On Fri, Dec 8, 2023, at 21:11, Even Rouault wrote: > >> >> - GeoTIFF DEFLATE 280 MB > Is it with PREDICTOR=3 ? (If not, then the ZARR datasets should also be > compressed with FILTER=DELTA to have a fair comparison) I would expect a fair comparison to be between PREDICTOR=3 and FILTER=DELTA

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Even Rouault via gdal-dev
 - GeoTIFF DEFLATE 280 MB Is it with PREDICTOR=3 ?  (If not, then the ZARR datasets should also be compressed with FILTER=DELTA to have a fair comparison)  - Zarr BLOSC zlib NONE 281 MB  - Zarr BLOSC zlib BIT 253 MB  - Zarr BLOSC zlib BYTE 249 MB -- http://www.spatialys.com My software is f

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Laurențiu Nicola via gdal-dev
Hi, In my (limited) experience, it can really make a difference for floating-point rasters. Testing with a small one I have on hand (10980x10980x1, Float32), I get: - GeoTIFF DEFLATE 280 MB - Zarr BLOSC zlib NONE 281 MB - Zarr BLOSC zlib BIT 253 MB - Zarr BLOSC zlib BYTE 249 MB Laurentiu

Re: [gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Even Rouault via gdal-dev
Jesse, This would break interoperability with other TIFF readers... Even adding a new TIFF tag to advertize that bit shuffling is applied would probably not be a sufficient guard, as existing readers wouldn't read it, and would just display garbage, which is worth that not being able to open

[gdal-dev] GTiff bit shuffle compression feature request

2023-12-08 Thread Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] via gdal-dev
Hi, When using horizonal differencing to reduce the numerical range of band data, the upper bytes in the produced stream are typically 0 which leverages LZ’s byte based compression model. But the least significant bytes can still have many significant bits as 0. Unless the whole byte is replic