Hi.
On 8/30/07, Martin Maechler <[EMAIL PROTECTED]> wrote:
> >>>>> "HenrikB" == Henrik Bengtsson <[EMAIL PROTECTED]>
> >>>>> on Wed, 29 Aug 2007 15:11:17 -0700 writes:
>
> >> From NEWS of R v2.6.0 devel:
> HenrikB> o R CMD check now does a recursive copy on the
> HenrikB> 'tests' directory.
>
> HenrikB> However, R CMD check does not run *.R scripts in
> HenrikB> such subdirectories (as I thought/hoped for),
>
> a pretty bold hope ;-)
>
> HenrikB> only those directly under tests/, This may or may not be
> HenrikB> intentional.
>
> intentional I'd say: I did not implement it, but it seems much
> more logical to keep the previous rule: All *.R files in
> ./tests/ are run <period>
> Subdirectories can be useful for organization, notably storing
> test data. I don't think it's a good idea to use so very many test files
> that you need subdirectories, unless maybe you are thinking
> about unit tests; and then, see below.
Examples of subdirectories (some overlapping) are:
units/ - tests of minimal code modules
integration/ - tests of integrating the above units
system/ - "real-world" scenarios/use cases
requirements/ - every requirement should have at least on test.
bugs/ - every bug fix should come with a new test.
regression/ - every update should have a regression test to validate
backward compatibility etc.
robustness/ - Testing the robustness of estimators against outliers as
well as extreme parameter settings.
validation/ - validation of numeric results compared with alternative
implementations or summaries.
benchmarking/ - actually more measuring time, but can involve
validation that a method is faster than an alternative.
crossplatform/ - validate correctness across platforms.
torture/ - pushing the limits.
>
>
> HenrikB> If true, maybe the above should be clarified as:
>
> HenrikB> o R CMD check now does a recursive copy on the
> HenrikB> 'tests' directory for the purpose of provided data
> HenrikB> files for input. Test scripts still has to be
> HenrikB> directly under tests/ to be run.
>
> HenrikB> Just a comment
>
> Thanks for the comment.
>
> Many other people have complained that 'NEWS' is too verbose
> already, and hence too much to be read by an average useR, even programmeR.
> The point about 'NEWS' is to mention things that are *changed*.
> Now since the behavior did *not* change, we definitely should
> not make NEWS longer just to say that.
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> A note about ./tests/ , directories, etc :
>
> At the useR!2007 meeting three weeks ago, there was an
> interesting discussion --- I think after the "Regulated
> Environments / R Certification" session {{and those interested
> there should look at http://www.r-project.org/certification.html}} ---
I ran in to that last week; I think that is a great and important
initiative. Now I hear that people use R for development and then
reimplement in SAS to please FDA.
> on the broader topic of 'R validation' and testing etc,
> where one of my short contributions was to acknowledge that
> "yes, it would be great if people wrote much more
> regression/unit test code, and I would welcome if these were
> collected, and put into daily testing state etc etc"
>
> *Contrary* to the <pkg>/tests/ concept,
> one thing I think we (R developers at large) should consider and
> often favor, is the idea that you want to run the tests also at
> run time and not just install time
> {{and part of that is also desirable for "base R"}}
>
> About a week agao, I have re-organized the
> RUnit-based unit regression tests for two Rmetrics R packages to
> achieve the following goals :
>
> 1) tests should be auto-run via 'R CMD check <pkg>'
> 2) tests should be easily re-runnable by a user who only
> got the *binary* package {e.g. a typical Windows user}
> 3) tests should also be easily runnable by the package developer
> without need to run the full R CMD check.
>
> where "1)" and "2)" where really primary goals.
>
> I've basically started from what is written (I think mainly by
> Tony Plate and Gregor Gorjanc) in the R Wiki section
> http://wiki.r-project.org/rwiki/doku.php?id=developers:runit
> and Gregor Gojanc's example in CRAN package 'gdata'
>
> but then really also found I needed to change things a bit to
> make '2)' more convenient.
>
> The basic idea is:
>
> ./tests/doRunit.R is a setup file {which will not be installed}
>
> ./inst/unitTests/ contains all the meat {and *is* installed}
> with
> ./inst/unitTests/runTests.R calls all the other unit tests
> ./inst/unitTests/runit*.R
>
> where the idea is the user of the *installed* package
> can easily run the tests again,
> hereby **validating** her installation of the package:
>
> E.g., for the not-yet released version of fCalendar,
>
> > source(system.file("unitTests/runTests.R", package = "fCalendar"))
>
> RUnit 0.4.17 loaded.
> Now have RUnit Test Suite 'testSuite' for package 'fCalendar' :
>
> ......
>
> Consider doing
> tests <- runTestSuite(testSuite)
>
> and later
> printTextProtocol(tests)
I think these are all great ideas/suggests/implementation. As
important as design and implementation, testing (and bug reporting)
deserves a higher status.
It often the case that it is not feasible to have all tests ran by R
CMD check. For instance, I have system tests that take literally
hours or even days to run. I also know that because of their
integrated build/check system, Bioconductor have a rule of thumb that
it shouldn't take more than 5 minutes for R CMD check to finish
[http://wiki.fhcrc.org/bioc/Package_Guidelines#time-requirements].
Then having test cases installed allows you to still include such
tests and run them at other times (run time).
I would like to add that it can also be useful to have the option tp
ask an end-user that reports bugs/problems to run some standard tests
(that might not have been ran by R CMD check).
Cheers
Henrik
>
> -- -- -- -- --
>
> The development version is available via subversion (svn) or also simple
> web tools from https://svn.r-project.org/Rmetrics/trunk/fCalendar/
>
> Martin Maechler, ETH Zurich
>
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
