On 1/1/2014, 12:50 PM, Jakub Jelinek wrote:
On Tue, Dec 31, 2013 at 03:22:07PM -0500, Vladimir Makarov wrote:
Scimark2 is always used by Phoronix to show how bad GCC in
comparison with LLVM. It is understandable. For some reasons
phoronix is very biased to LLVM and, I'd say, a marketing machine
for LLVM.
They use very small selected benchmarks. Benchmarking is evil but I
believe more SPEC2000/SPEC2006 which use bigger real programs. With
their approach, they could use whetstone instead of Scimark but the
results would be very bad for LLVM (e.g. 64-bit code for -O3 on
Haswell):
I'm aware of all that, that said putting some minimal effort to increase
obstackles for their marketing is worth it.
I believe that Michael Larabel has no idea what he is measuring.
I don't believe in his data too. Recently I tried to reproduce Scimark
results for GCC4.8 and LLVM3.3 from his article "LLVM Clang 3.4 Already
Has Some Performance Changes":
http://www.phoronix.com/scan.php?page=article&item=llvm_clang34_first&num=2
He used i7-4770K for this. I used the closest machine I found i5-4670
(with switched turbo mode off). The important difference is 0.1Ghz in
frequency (3.5Ghz vs. 3.4 Ghz). I got close GCC Scimark (-large)
composite score when I used -O and still on my machine the composite
score was 20% higher than the article reports although the article says
that -O3 -march=core-avx were used.
His articles about LLVM and GCC usually contains a lot of negative
emotions about GCC and positive ones about LLVM. I don't know may be he
tried to communicate with GCC community and got a frustrated experience.
That is only my explanation now why he is so biased towards LLVM.
So far I saw several pitfalls in his testing methodology:
o Micro-benchmarking. You just gave an example LU-factorization in
Scimark2 where most benchmark time is spent in 2-lines loop.
o Comparing LLVM and GCC on fortran benchmarks. LLVM has no fortran FE
and just quietly call system GCC. So comparison LLVM and GCC on fortran
benchmarks means comparison system GCC and a given GCC.
o IMO, the data in articles lack credability may be because a wrong setup.
