-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 04/16/2013 07:27 PM, Paul Coverdale wrote:
> Yes, I think there is still some work to be done on the plan, I
> noticed for example that input level and listening level still need
> to be defined. And why are you using MUSHRA? It may be quite
> time-consuming with a large number of conditions.
MUSHRA is actually the least time-consuming and it is designed for
"intermediate quality level", which is mostly what we're after here.
Technically the 12 kbps content may be in the MOS area and the 96 kb/s
content would be in the BS.1116-1 area, but I suspect MUSHRA is
probably goo enough for everything.
> But just to comment on some of the points raised by Jean-Marc: - we
> need to be consistent about the input filtering. We can't use
> different bandwidths for Opus and AMR-WB.
Well, then make it 8 kHz for everything. There's no point in degrading
the input on purpose. In fact, technically it should be 48 kHz and
then each codec gets to decide what it does to it.
> - I don't understand the point about making sure to tell the Opus
> encoder what the percentage of loss is so it can optimize the
> encoding for it. What happens in practice when the loss percentage
> isn't known?
Well, that's what RTCP and others are for. Sure you don't necessarily
know that you have 4.9% or 5% loss, but you can have a pretty good idea.
Jean-Marc
>
> Cheers,
>
> ...Paul
>
>
>> -----Original Message----- From: [email protected]
>> [mailto:[email protected]] On Behalf Of Jean-Marc Valin
>> Sent: Tuesday, April 16, 2013 2:36 PM To: Christian Hoene Cc:
>> [email protected]; [email protected];
>> [email protected] Subject: Re: [codec] Opus
>> comparision test plan
>>
>> Christian,
>>
>> I was only able to look at your plan quickly but still found many
>> issues with it. I think you should take your time to fix those
>> and also all the ones I'm sure I missed. So far...
>>
>> == AMR-WB test ==
>>
>> - You should use 8 kHz bandwidth for Opus. Opus it not restricted
>> to 7 kHz and low-pass filtering will hurt quality. - For AMR-WB
>> the 8.85 mode is "intended to be used only temporarily during
>> severe radio channel conditions or during network congestion",
>> so you should probably test the 18.25 kbps mode instead. - It's
>> not clear from your document, but Opus should use VBR - If you're
>> going to test packet loss, there should be at least one test with
>> FEC -- probably the 23.85 one - Make sure to tell the Opus
>> encoder what the percentage of loss is so it can optimize the
>> encoding for it.
>>
>> == AAC-ELD test ==
>>
>> - Some of the rates in the test are in the bitrates where both
>> codecs are expected to be "falling apart", so we will not learn
>> much there (really bad quality vs absolutely terrible doesn't
>> help because people don't want either). - Some of the rates are
>> way too high for listeners to be able to reliably judge. These
>> are guaranteed to end up with everything tied unless there's a
>> problem with the test. - Using SBR at 128 kb/s for ELD is useless
>> and probably harming quality - There is no reason to test music
>> at 24/32 kHz because music is almost always available at 48 kHz,
>> so it would hurt to artificially limit the signal. - When
>> considering CBR vs VBR you have to consider the delay, but also
>> the different meanings these have between codecs. I'm not
>> familiar with the details, but what most codecs like AAC-LC and
>> MP3 call "CBR" actually uses a bit reservoir and is equivalent to
>> Opus "Constrained VBR". - The choice of music samples is
>> critical. Just that choice can sometimes determine the outcome of
>> a music test. You need a wide variety of samples to get something
>> statistically significant. It's also probably a good idea to use
>> "normal" samples at low rates and "hard" samples at high rate. -
>> I'd need to recheck the MUSHRA spec, but I'm pretty sure mixing
>> mono and stereo in the same session is highly discouraged. - Not
>> sure in what form, but it might be worth testing the pre-1.1
>> encoder of Opus.
>>
>> In general, for mono music, I'd recommend restricting yourself to
>> rates between 40 and 64 kb/s. For stereo music, I'd stay within
>> 56-96 kb/s. Outside of this you're either in the falling apart
>> region or in the "nearly transparent for most listeners".
>>
>> Also, I hope you will fix your methodology from the previous test
>> to actually tell the listeners about the hidden reference.
>>
>> In any case, this is only what I've been able to find with a
>> quick look. It's probably good to have wider review once you fix
>> everything.
>>
>> Jean-Marc
>>
>> On 04/16/2013 03:43 AM, Christian Hoene wrote:
>>> Hello,
>>>
>>> as for next week, we place to compare Opus against AMR-WB (with
>>> noisy signals) and AAC-eLD. Attached our preliminary test plan.
>>> Feel free to criticize it.
>>>
>>> In addition to the comparison, we will do some Opus
>>> characterization (Opus vs. Opus). More to about this test
>>> later.
>>>
>>> With best regards,
>>>
>>> Christian Hoene
>>>
>>>
>>>
>>>
>>> Symonics GmbH Sand 13 72076 Tübingen Tel +49 7071 5681302 Fax
>>> +49 7071 5681309 Email: [email protected]
>>>
>>> Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene,
>>> Patrick Schreiner Sitz der Gesellschaft/Place of Business:
>>> Tübingen Registereintrag/Commercial Register: Amtsgericht
>>> Stuttgart, HRB 739918
>>>
>>>
>>>
>>>
>>> _______________________________________________ codec mailing
>>> list [email protected]
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>
>> _______________________________________________ codec mailing
>> list [email protected] https://www.ietf.org/mailman/listinfo/codec
>
> _______________________________________________ codec mailing list
> [email protected] https://www.ietf.org/mailman/listinfo/codec
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJRbe93AAoJEJ6/8sItn9q9tqQH/0mafc0HRPKju+frdqv+NTOY
LIITkewiczNnKPX2XKv6lZJ0YS81rJCGo+QJfRayCbXgPo8G/5B9Zdl1PF5NatYx
yAtS6PBkNUC+jSJQZ72YNEXxwJhoidGflFK52+MH/eqe1jlWxgzDPEk+gLcjG31c
8a6U89PCgeaZE98g+/wTj0RGOCo03/5nj8jRi8l23HN/y14B4ghkpzQE3/VG/4X0
deWz+NzjtdEw+zjrbwivaRQyrtndjDg7UfXcB8Lks7CWlIvwoDIViN2t2iyg+m+D
yk1deJSbQiyOoGT5V1VcSTyV7xk6azuTkNW4drNUXN1DjfuEYdHRkZ6IHm7FpfE=
=US/j
-----END PGP SIGNATURE-----
_______________________________________________
codec mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/codec
