You input in valued.
For speed/size I was think we need to have sanity checks as gcc versions
have been known to suddenly dump a bunch of unexpected data in the
middle of a linked program.
Of course all the functional tests still give correct results and all
looks good!
The second use would be as evaluating impact of a patch - so we can see
that if it causes significant degradations in other areas. Qualative
perhaps - but the alternative is nothing..
gcc test suite has full set of answers and framework. We may not pass
all the marked applicable tests due to limited memory or unimplemented
functions. Yet even with this functional regression can easily be seen.
Andy
Dave N6NZ wrote:
Andrew Hutchinson wrote:
Here my input:
For starters gcc has testsuite that can be used. It's not perfect but
its quite demanding - even if we cant do all the tests.
Yes, we could probably pull a subset of meaningful test from that
which would give us a great start. The hard/tedious part will be
generating our expected results.
After than I suggest some "benchmark" that would produce more normal
code and also give qualitative indications of performance (size is
easy, speed would be nice).
Finally, regression tests using testcases and bug reports.
Performance regressions are particularly nasty to test. First you have
to have code that can sensitize an optimization that you want. Then,
you need to have some expected results. The compare of actual to
expected is difficult -- what are you measuring? And how close is
good enough? Exactly clock count? Clock count within a guard band?
A specific assembler sequence that must happen (but actual registers
used don't matter)? Some registers matter?
Then you have the test matrix... what is the expected result with -Os
versus -O3? etc, etc, etc.
All this presumes a test framework that can capture all the required
outputs to check against expected results: .S, clock count, code size,
exit code,.... Oh, and getting the right numerical (or whatever)
answer... and are we getting it for the right reason? The compiler
might show a great speed and code size improvement by mistakenly
ignoring the volatile keyword in some optimization, and a simulator
might never show a wrong answer :(
Having been purely a gcc and avr-gcc user, I don't have any idea if
any parts of this framework already exist in a form that addresses
these problems.
Anyway, the test matrix is huge, and a simulator is going to limit the
ultimate performance of the test rig. So we won't be able to do
everything we can think of.
On the plus side of the ledger: Our audience, being embedded
developers, have very specific needs, so that can help prioritize.
OK, this e-mail was a shotgun blast of issues completely devoid of
specifics :) :)
Andy
-dave
Dave N6NZ wrote:
Well, after spamming the wrong list with this, I hope I've got the
right place now :(
-dave
------------------------------------------------------------------------
Subject:
Re: [avr-gcc-list] GCC-AVR Register optimisations
From:
Dave N6NZ <[EMAIL PROTECTED]>
Date:
Thu, 10 Jan 2008 12:35:10 -0800
To:
gEDA user mailing list <[EMAIL PROTECTED]>
To:
gEDA user mailing list <[EMAIL PROTECTED]>
Weddington, Eric wrote:
- I, and others, are very interested in what you are using to test
your
proposed changes. I have plans to put together an AVR Benchmark Suite,
consisting of a variety of publicly available programs that can be
used
to test the compiler performance over time. It definitely needs to
have
different types of programs, and publicly available programs so there
are no issues with distributing such a Benchmark Suite. I welcome any
collaboration on this.
Hi,
Over the course of years I have worked in and managed various
different hardware and software validation teams. This is probably
a good way for me to contribute, since I'm not a compiler back-end
guru, but I have relevant experience writing and reviewing test
plans. Some of those test plans even targeted C compilers :)
-dave
------------------------------------------------------------------------
_______________________________________________
AVR-GCC-list mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
_______________________________________________
AVR-GCC-list mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
_______________________________________________
AVR-GCC-list mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
