Hi Greg, On Wed, 1 Sept 2021 at 01:33, Greg Troxel <g...@lexort.com> wrote:
> A few comments from the perspective of a gdal test newbie, which I hope > are helpful: > > * autotest is not in the distribution tarball > Yeah, it's never been in the release tarballs. And the release tarballs start one folder level down (gdal/) as well. TBH I think the release tarballs should just match/be an archive of the git tag. But that's above my pay grade :-) @Debian and other package maintainers — are you just getting the Git tags these days or do you actually use the release tarballs? > > + Does it test the gdal that is installed in one's PATH (and > PYTHONPATH), or is is testing the gdal that is in the source tree? > CONTRIBUTING.md has a guide to run `scripts/setdevenv.sh` which sets up in-tree execution for utilities/libraries and the tests. > > + Is one supposed to check out the corresponding branch (or really > tag), or is it ok to test 3.3.2rc2 with master? > The tests should match the code, so 3.3.2rc2 tests run against 3.3.2rc2 code. > * no clear path to test a not-yet-installed gdal > > When building a new gdal, it seems a good idea to run tests before > installing it. autoconf has done this for year with make check. I see > no obvious way to do this. > setdevenv.sh? Adding some sort of `make test` might be a cleaner approach though. > > * puzzling test dependencies > > It's not really clear if a missing dependency (vs requirements.txt) > will cause problems (assuming pytest and lxml are there), in terms of > just skipping some things, bad output, or perhaps some missing pretty > printing. > a mixture. Is there some particular reason installing everything in requirements.txt from PyPi is a problem? > I think pytest-env is > https://pypi.org/project/pytest-env/#history > but that appears unmaintained, with a last release in 2017. Perhaps > there are no outstanding bugs and no changes and it is actually > maintained but there is no reason to change -- but that is highly > implausible. It seems boutique, based on not being in pkgsrc, which > means no one else has needed it, even though we have 20K packages. > It's about 45 lines of code. Pytest plugins are often fairly minimal, and just do One Thing. > [Now I see that osr tests fail with a warning about env if this is > missing, but really the test should be skipped or hard fail and not > run without the env.] > Yes, I think it should hard-fail > * I'm guessing the expectation is that all the tests pass. With > autotest from master and 3.3.2rc2 installed (base lib and python > bindings), I get: > In general all the tests pass. There's skips/xfails for particular known issues, but maybe you've found a bug? Maybe BSD-specific, BSD isn't part of CI. > > * skipping tests after a failure. > It would be nice to have hints in README.md about skipping tests, but it > seems easy enough to run osr and then the following ones. > you mean, stopping on the first failure? `pytest -x ...` autotest/README.md has some docs on running specific tests / modules / etc GDAL has some odd test layouts with particular inter-test dependencies from when the test suite was bulk-ported via automation to work under pytest — this made it a lot saner, but some of the "test 18 depends on test 17 passing" issues remain. Rob :) > >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
