Oh that’s so weird. Well that puts my take-away as: just use Level=17 and forget the rest exist!
-Matt From: Daniel Evans <daniel.ev...@jbarisk.com> Sent: March 15, 2021 2:36 AM To: Matt.Wilkie <matt.wil...@yukon.ca>; gdal-dev@lists.osgeo.org Subject: RE: ZSTD compression level 17 smaller than 18+? *** External email: Do not click on links or attachments except from trusted senders. *** ****************************************************************************************** I don’t think it’s particularly unusual for any compression algorithm to have this sort of behaviour once it’s “maxed out”. Running the in-built zstd benchmarks from the command line without any custom parameters – so entirely separate of how GDAL uses it – it shows your observed behaviour to an even greater extent, with most levels doing worse than level 1! Level Size % Smaller than Level 1 1 3154783 0.00% 2 3130807 0.76% 3 3231415 -2.43% 4 3346308 -6.07% 5 3335646 -5.73% 6 3340284 -5.88% 7 3324298 -5.37% 8 3315170 -5.08% 9 3321085 -5.27% 10 3355473 -6.36% 11 3355932 -6.38% 12 3355857 -6.37% 13 3355401 -6.36% 14 3354678 -6.34% 15 3353801 -6.31% 16 3083731 2.25% 17 3142331 0.39% 18 3147466 0.23% 19 3146084 0.28% 20 3146548 0.26% 21 3145580 0.29% 22 3142972 0.37% Table generated from output of `zstd -b1 -e22`, “zstd command line interface 64-bits v1.4.7”. Dr Daniel Evans Software Developer From: gdal-dev <gdal-dev-boun...@lists.osgeo.org<mailto:gdal-dev-boun...@lists.osgeo.org>> On Behalf Of matt.wil...@yukon.ca<mailto:matt.wil...@yukon.ca> Sent: 15 March 2021 04:04 To: gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> Subject: [gdal-dev] ZSTD compression level 17 smaller than 18+? So this is curious: output to Cloud optimized Geotiff with ZSTD compression to Level 17 is smaller than levels 18-22. Or at least that's my results when testing compression results with Gdal 3.1 via Qgis 3.18 on Windows 10. I expected compression levels to get smaller with higher levels, at expense of longer and longer computation time. Levels 1 to 16 do that. Even more curious than 17 being the smallest of the lot is that each step 18 to 22 is progressively larger, albeit by a miniscule amount. Here's my results: % smaller than biggest Bytes Filename Compression Predictor Level 24.2% 200,968,821 cog-zstd-pyes-lev17.tif zstd yes 17 22.8% 204,567,155 cog-zstd-pyes-lev18.tif zstd yes 18 22.8% 204,594,678 cog-zstd-pyes-lev19.tif zstd yes 19 22.8% 204,594,678 cog-zstd-pyes-lev20.tif zstd yes 20 22.8% 204,626,722 cog-zstd-pyes-lev21.tif zstd yes 21 22.8% 204,635,985 cog-zstd-pyes-lev22.tif zstd yes 22 22.6% 205,026,301 cog-zstd-pyes-lev16.tif zstd yes 16 21.5% 208,039,012 cog-zstd-pyes-lev15.tif zstd yes 15 21.5% 208,094,559 cog-zstd-pyes-lev13.tif zstd yes 13 21.5% 208,094,560 cog-zstd-pyes-lev14.tif zstd yes 14 21.3% 208,714,731 cog-zstd-pyes-lev12.tif zstd yes 12 21.2% 208,989,301 cog-zstd-pyes-lev11.tif zstd yes 11 21.2% 208,989,994 cog-zstd-pyes-lev10.tif zstd yes 10 21.1% 209,071,549 cog-zstd-pyes-lev9.tif zstd yes 9 21.0% 209,453,858 cog-zstd-pyes-lev8.tif zstd yes 8 20.9% 209,634,360 cog-zstd-pyes-lev7.tif zstd yes 7 20.7% 210,182,178 cog-zstd-pyes-lev6.tif zstd yes 6 20.4% 210,992,074 cog-zstd-pyes-lev5.tif zstd yes 5 18.9% 214,894,175 cog-zstd-pyes-lev4.tif zstd yes 4 17.4% 218,911,762 cog-zstd-pyes-lev3.tif lzw 2 3 16.6% 220,991,236 cog-dflt-p2.tif deflate 2 13.0% 230,553,902 cog-zstd-pyes-lev2.tif zstd yes 2 11.6% 234,277,147 cog-zstd-pyes-lev1.tif zstd yes 1 5.7% 249,979,858 cog-dflt-p1.tif deflate 1 0.0% 265,062,854 cog-lzw-p2.tif lzw 2 Command line pattern: gdal_translate ^ -co compress=zstd ^ -co predictor=yes ^ -co level=3 ^ -co num_threads=all_cpus ^ -of cog ^ mtn-view.tif ^ mtn-view-cog-zstd-pyes-lev3.tif Source image: Name AerialPhotos/AerialPhotos URL https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer Source crs='EPSG:3578' format='JPGPNG' layer='' url='https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer' CRS EPSG:3578 - NAD83 / Yukon Albers - Projected Export extent Upper Left, Lower Left, Upper Right, Lower Right 356750.816, 702590.026 356750.816, 699097.526 361572.583, 702590.026 361572.583, 699097.526 Pixel size 0.5m Do others see the same thing? Cheers, Matt Wilkie Environment | Information Management & Technology | 867-667-8133 🌄 Yukon.ca<http://yukon.ca/> T +44 (0)1756 799919 www.jbarisk.com<http://www.jbarisk.com> [Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png] [Follow us on Twitter] <https://twitter.com/jbarisk> Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT> The JBA Group supports the JBA Trust. All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
