I don't know why you're pouring scorn on this exercise, Ron. It seems to me that it is a bona-fide attempt to understand the strengths and weaknesses of the Opus codec in a controlled, unbiased manner, what a characterisation test should do. It should have been done as part of the IETF codec WG activity, but better late than never.
Cheers, ...Paul >-----Original Message----- >From: [email protected] [mailto:[email protected]] On Behalf >Of Ron >Sent: Tuesday, April 23, 2013 3:31 PM >To: [email protected] >Cc: [email protected] >Subject: Re: [codec] Audio tests: Further steps > >On Tue, Apr 23, 2013 at 09:50:16AM +0200, Christian Hoene wrote: >> Hi, >> >> currently, the codec comparison tests are running. Because of the >> request of many codec developers, we plan to extend those tests: We >> might add audio tests in which the content is varied to a large >> extend. For that, we need sample that cannot be compressed well by >> Opus or AAC-eLD. For me, it is easy to get those difficult samples for >> Opus. It is much challenging to get those for AAC-eLD. Thus, if >> somebody had to time to study the weaknesses of AAC-eLD, please >forward me the samples. > >Uhm, so ... while I'm certain that the codec developers will be >delighted if you can point out any new killer samples that they aren't >yet aware of (since significant work has already been made to improve >the encoder for the known ones, and that work is still ongoing) -- I'm >also pretty certain that going out of your way to deliberately select >such samples immediately disqualifies this from being characterised as a >"comparison test", or at least claiming that it's even remotely >representative of what people will observe over a general corpus of >their own audio, given the degree to which such samples really are >outliers. > >> I cannot start fair tests if I do not have challenging samples for >> both codecs. > >While such a test might have some novelty value to show "here are some >non-exhaustive results for the worst samples that we could find in a few >days of searching", I'm pretty sure words like "fair" and "scientific >rigour" don't really belong in the same sentence. Not in the least when >you also say "we have the established list for one codec, but the known >killers for the other is at present entirely unknown to us". > >If you want to spend your time doing that, that's fine, and the results >may well be 'interesting'. But mischaracterising them as a "comparison" >test would just be somewhere on the spectrum from "mildly amusing" to "a >sad day for Modern Science". > >It's your reputation though, and I can't tell you how to spend it. >But you might want to think this through a little better if you are >going to paint this with the brush of Being Science. > >This isn't the cosmetics industry, other people can measure these things >too, and will continue to for some time to come. > > Cheers, > Ron > > >_______________________________________________ >codec mailing list >[email protected] >https://www.ietf.org/mailman/listinfo/codec _______________________________________________ codec mailing list [email protected] https://www.ietf.org/mailman/listinfo/codec
