[issue22881] show median in benchmark results

Changes by Stefan Behnel : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue22881] show median in benchmark results

STINNER Victor added the comment: The new https://github.com/python/performance benchmark suite now displays the median rather than the arithmetic mean (average) by default (it also displays the standard deviation). See the perf issue for a discussion about median vs mean: https://github.com/h

[issue22881] show median in benchmark results

Antoine Pitrou added the comment: > May be just drop 5% of largest values to avoid the impact of outliers? In fast mode (option "-f"), there may not be enough samples for that. -- ___ Python tracker __

[issue22881] show median in benchmark results

Stefan Behnel added the comment: Well, we can apply a kludge, or apply statistics. -- ___ Python tracker ___ ___ Python-bugs-list mail

[issue22881] show median in benchmark results

Serhiy Storchaka added the comment: May be just drop 5% of largest values to avoid the impact of outliers? See also issue23552. -- ___ Python tracker ___ ___

[issue22881] show median in benchmark results

Antoine Pitrou added the comment: Then let's just replace the average with the median? I don't think it makes sense to add more statistical information to the output (IMHO, there is already too much of it :-)). -- ___ Python tracker

[issue22881] show median in benchmark results

Stefan Behnel added the comment: I'm actually not sure how it relates to the minimum. The more runs you have, the higher the chance of hitting the actual minimum at least once. And if none of the runs hits the real minimum, you're simply out of luck. However, it should tend to give a much bett

[issue22881] show median in benchmark results

Antoine Pitrou added the comment: Have you found the median to be more stable than the minimum here? -- ___ Python tracker ___ ___ Pyt

[issue22881] show median in benchmark results

Stefan Behnel added the comment: > for the even number case, I think you shouldn't do // 2, but / 2. Right. I updated the patch. -- Added file: http://bugs.python.org/file39279/show_median.patch ___ Python tracker

[issue22881] show median in benchmark results

Wolfgang Maier added the comment: ah sorry, it's late here already and I forgot what file this change is about. So forget my last comment then. -- ___ Python tracker ___ ___

[issue22881] show median in benchmark results

Wolfgang Maier added the comment: > It's not available in older Python versions, e.g. 2.6. I know, I was talking about 3.5+, of course. This would not be backported to Python2 anyway, would it? -- ___ Python tracker

[issue22881] show median in benchmark results

Stefan Behnel added the comment: > In general, wouldn't it be good to let the statistics module do all the stats > calculations? It's not available in older Python versions, e.g. 2.6. -- ___ Python tracker __

[issue22881] show median in benchmark results

Wolfgang Maier added the comment: for the even number case, I think you shouldn't do // 2, but / 2. In general, wouldn't it be good to let the statistics module do all the stats calculations? -- ___ Python tracker

[issue22881] show median in benchmark results

Changes by Wolfgang Maier : -- nosy: +wolma ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue22881] show median in benchmark results

Stefan Behnel added the comment: Any more comments on the patch, or can it be applied? -- ___ Python tracker ___ ___ Python-bugs-list

[issue22881] show median in benchmark results

Stefan Behnel added the comment: Fair enough, patch updated. -- Added file: http://bugs.python.org/file37207/show_median.patch ___ Python tracker ___

[issue22881] show median in benchmark results

Serhiy Storchaka added the comment: In case of even number of samples the median value is calculated as arithmetic mean of two middle samples. med_base = (base_times[len(base_times)//2] + base_times[(len(base_times)-1)//2]) / 2 -- ___ Python tracke

[issue22881] show median in benchmark results

Changes by Serhiy Storchaka : -- Removed message: http://bugs.python.org/msg231240 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue22881] show median in benchmark results

Serhiy Storchaka added the comment: In case of even number of samples the median value is calculated as arithmetic mean of two middle samples. med_base = (base_times[len(base_times)//2] + base_times[(len(base_times)+1)//2]) / 2 -- nosy: +serhiy.storchaka stage: -> patch review _

[issue22881] show median in benchmark results

New submission from Stefan Behnel: The median tends to give a better idea about benchmark results than an average as it inherently ignores outliers. -- components: Benchmarks files: show_median.patch keywords: patch messages: 231239 nosy: pitrou, scoder priority: normal severity: normal