On 01/30/2018 09:50 AM, James E Keenan wrote:
On 01/28/2018 04:36 PM, Andrew Hewus Fresh wrote: [snip] > > I think this is OK, although I've been led to believe that we usually > prefer to enable as many tests as possible so that we don't get > different results depending on what folks might have installed or what > environment they have set, which would mean making these additions. > > TEST_ENV = AUTOMATED_TESTING=1 > TEST_DEPENDS = devel/p5-Test-MockObject \ > devel/p5-Test-Pod>=1.00 \ > devel/p5-Test-Pod-Coverage>=0.08 \ > devel/p5-Test-CPAN-Meta > > # Additional tests could be enabled by porting > # devel/p5-Test-CPAN-Meta-JSON >Andrew,I agree with you that, in general, we don't want to get different test results depending on what particular users have installed on their systems. That would lead us to activate tests which the CPAN author indicates can be run if certain other Perl extensions are present.However, I think we should look more closely at the environmental variable which you are suggesting to include in the port's Makefile:##### TEST_ENV = AUTOMATED_TESTING=1 #####In the context of tests in CPAN distributions 'AUTOMATED_TESTING' has a specific meaning. At the 2013 Perl QA Hackathon held in Lancaster, U.K., a consensus emerged to define the meaning of certain terms that would be used by CPAN authors to activate or not activate tests under different circumstances (https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md#environment-variables-for-testing-contexts). Here's what was recommended for 'AUTOMATED_TESTING':#####AUTOMATED_TESTING: if true, tests are being run by an automated testing facility and not as part of the installation of a module; CPAN smokers must set this to true; CPAN clients must not set this#####In this context, "automated testing facility" refers to a server where someone is testing large parts of the 36,000+ distributions on CPAN against various versions of Perl on a specific platform and is sending test reports to cpantesters.org. For example, Alceu Rodrigues de Freitas Junior runs CPAN smokers for testing distributions against Perl 5.26.0, 5.26.1 and 5.27.6 (blead) on OpenBSD 6.0, 6.1 and 6.2. In those set-ups, he has indeed turned on an AUTOMATED_TESTING environmental variable; see, e.g., http://www.cpantesters.org/cpan/report/7f3fb1ae-e729-11e7-8087-d3e9cf4e8a6e.However, the Lancaster consensus guidance above indicates that in all circumstances other than preparing reports for cpantesters.org, AUTOMATED_TESTING should not be turned on. An OpenBSD port of a CPAN distribution is, essentially, a set of instructions for fetching a tarball from CPAN and installing it on OpenBSD. It is therefore a "CPAN client" as defined by the Lancaster consensus and, hence, should *not* turn on the AUTOMATED_TESTING environmental variable.The author of Devel-Platform-Info, Barbie, has participated in many Perl QA hackathons and was the administrator of cpantesters.org for many years. Hence, we should presume that he is using 'AUTOMATED_TESTING' in the way described above. Here's how Barbie structures his tests:##### # From:t/90podtest.tplan skip_all => "Author tests not required for installation" unless ($ENV{AUTOMATED_TESTING});eval "use Test::Pod 1.00"; plan skip_all => "Test::Pod 1.00 required for testing POD" if $@; #####He characterizes the tests in this file as "[a]uthor tests not required for installation" which are to be run only if the AUTOMATED_TESTING envvar is set. I interpret this to mean that they should only be run in the context of CPAN smokers. He then goes on to say that the tests in this file should only be run if Test::Pod version 1.00 or greater is already present on the user's machine.If the first 'plan skip_all' statement were not present in this file, then the author would be telling the user, "Go ahead and run this file if you've already got Test::Pod installed, but don't bother if you don't have it installed." If that were the case, then I think it would be appropriate for an OpenBSD port's Makefile to include the port corresponding to Test::Pod as a value for TEST_DEPENDS. But that's not the case here. The author is indicating that the Test::Pod tests should only be run by CPAN smokers, not by CPAN clients. If a CPAN client turns on AUTOMATED_TESTING, it is, in effect, second-guessing the author.Now I concede that:* The CPAN author may be incorrect; hence he or she should be second-guessed.* OpenBSD is free to set policies for ports that adhere to standards different from those set by the authors of the code being ported.* I'm just a beginner with respect to OpenBSD ports, so I may not know the historical context.So, I'm perfectly willing to revise the port's Makefile per your suggestion. But it would be good to have an explicit and well-reasoned policy to which I and other porters can conform going forward.Thank you very much. Jim Keenan
Attached please find a tarball revised per previous suggestions, i.e., not taking into account the questions I raised.
Thank you very much. Jim Keenan
p5-List-Compare.R1.tar.gz
Description: application/gzip
