On 01/28/2018 04:36 PM, Andrew Hewus Fresh wrote:
[snip]
>
> I think this is OK, although I've been led to believe that we usually
> prefer to enable as many tests as possible so that we don't get
> different results depending on what folks might have installed or what
> environment they have set, which would mean making these additions.
>
> TEST_ENV = AUTOMATED_TESTING=1
> TEST_DEPENDS = devel/p5-Test-MockObject \
> devel/p5-Test-Pod>=1.00 \
> devel/p5-Test-Pod-Coverage>=0.08 \
> devel/p5-Test-CPAN-Meta
>
> # Additional tests could be enabled by porting
> # devel/p5-Test-CPAN-Meta-JSON
>
Andrew,
I agree with you that, in general, we don't want to get different test
results depending on what particular users have installed on their
systems. That would lead us to activate tests which the CPAN author
indicates can be run if certain other Perl extensions are present.
However, I think we should look more closely at the environmental
variable which you are suggesting to include in the port's Makefile:
#####
TEST_ENV = AUTOMATED_TESTING=1
#####
In the context of tests in CPAN distributions 'AUTOMATED_TESTING' has a
specific meaning. At the 2013 Perl QA Hackathon held in Lancaster,
U.K., a consensus emerged to define the meaning of certain terms that
would be used by CPAN authors to activate or not activate tests under
different circumstances
(https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md#environment-variables-for-testing-contexts).
Here's what was recommended for 'AUTOMATED_TESTING':
#####
AUTOMATED_TESTING: if true, tests are being run by an automated testing
facility and not as part of the installation of a module; CPAN smokers
must set this to true; CPAN clients must not set this
#####
In this context, "automated testing facility" refers to a server where
someone is testing large parts of the 36,000+ distributions on CPAN
against various versions of Perl on a specific platform and is sending
test reports to cpantesters.org. For example, Alceu Rodrigues de
Freitas Junior runs CPAN smokers for testing distributions against Perl
5.26.0, 5.26.1 and 5.27.6 (blead) on OpenBSD 6.0, 6.1 and 6.2. In those
set-ups, he has indeed turned on an AUTOMATED_TESTING environmental
variable; see, e.g.,
http://www.cpantesters.org/cpan/report/7f3fb1ae-e729-11e7-8087-d3e9cf4e8a6e.
However, the Lancaster consensus guidance above indicates that in all
circumstances other than preparing reports for cpantesters.org,
AUTOMATED_TESTING should not be turned on. An OpenBSD port of a CPAN
distribution is, essentially, a set of instructions for fetching a
tarball from CPAN and installing it on OpenBSD. It is therefore a "CPAN
client" as defined by the Lancaster consensus and, hence, should *not*
turn on the AUTOMATED_TESTING environmental variable.
The author of Devel-Platform-Info, Barbie, has participated in many Perl
QA hackathons and was the administrator of cpantesters.org for many
years. Hence, we should presume that he is using 'AUTOMATED_TESTING' in
the way described above. Here's how Barbie structures his tests:
#####
# From:t/90podtest.t
plan skip_all => "Author tests not required for installation" unless
($ENV{AUTOMATED_TESTING});
eval "use Test::Pod 1.00";
plan skip_all => "Test::Pod 1.00 required for testing POD" if $@;
#####
He characterizes the tests in this file as "[a]uthor tests not required
for installation" which are to be run only if the AUTOMATED_TESTING
envvar is set. I interpret this to mean that they should only be run in
the context of CPAN smokers. He then goes on to say that the tests in
this file should only be run if Test::Pod version 1.00 or greater is
already present on the user's machine.
If the first 'plan skip_all' statement were not present in this file,
then the author would be telling the user, "Go ahead and run this file
if you've already got Test::Pod installed, but don't bother if you don't
have it installed." If that were the case, then I think it would be
appropriate for an OpenBSD port's Makefile to include the port
corresponding to Test::Pod as a value for TEST_DEPENDS. But that's not
the case here. The author is indicating that the Test::Pod tests should
only be run by CPAN smokers, not by CPAN clients. If a CPAN client
turns on AUTOMATED_TESTING, it is, in effect, second-guessing the author.
Now I concede that:
* The CPAN author may be incorrect; hence he or she should be
second-guessed.
* OpenBSD is free to set policies for ports that adhere to standards
different from those set by the authors of the code being ported.
* I'm just a beginner with respect to OpenBSD ports, so I may not know
the historical context.
So, I'm perfectly willing to revise the port's Makefile per your
suggestion. But it would be good to have an explicit and well-reasoned
policy to which I and other porters can conform going forward.
Thank you very much.
Jim Keenan
