I have a couple of fundamental issues with how you're calculating 3 of the 4 metrics (all but RGB-SSIM, which I didn't think too much about)
First, am I correct in my reading of your methodology that for all metrics, you encode a color image (4:2:0) and use that encoded filesize? If so, then all the results from greyscale metrics are invalid, as the filesize would include chroma, but the metric only measures luma. An encoder could spend 0 bits on chroma and get a better score than an encoder that spent more bits on chroma than luma. Second, for Y-SSIM and IW-SSIM only, it appears you encode as color, then afterwards convert both the original image and encoded image to greyscale and calculate SSIM between those two images. This is fundamentally wrong - the original converted to greyscale was not the image the codec encoded, so you're not measuring the distortion of the codec. It looks like PSNR-HVS-M is calculated from the YUV fed into the encoder, which is how Y-SSIM and IW-SSIM should be calculated as well. Fortunately, the solution is easy - for greyscale metrics, simply convert to greyscale before encoding, not after. Or, if that's what you're already doing, make it clear. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
