On 05/04/2018 07:57 PM, Dylan Baker wrote:
Quoting Juan A. Suarez Romero (2018-05-04 04:50:27)On Fri, 2018-05-04 at 12:03 +0200, Juan A. Suarez Romero wrote:On Wed, 2018-05-02 at 13:57 -0700, Dylan Baker wrote:Quoting Juan A. Suarez Romero (2018-05-02 09:49:08)Hi, Dylan.I see you've pushed this series. Now, when I'm trying to run some profiles (mainly, tests/crucible and tests/khr_gl* ), seems they are broken: [0000/7776] Traceback (most recent call last): File "./piglit", line 178, in <module> main() File "./piglit", line 174, in main sys.exit(runner(args)) File "/home/igalia/jasuarez/piglit/framework/exceptions.py", line 51, in _inner func(*args, **kwargs) File "/home/igalia/jasuarez/piglit/framework/programs/run.py", line 370, in run backend.finalize({'time_elapsed': time_elapsed.to_json()}) File "/home/igalia/jasuarez/piglit/framework/backends/json.py", line 163, in finalize assert data['tests'] AssertionError J.A.Dang. I can't reproduce any failures with crucible, though I did make it thread safe and fix the using a config file :) I can't get the glcts binary to run, no matter what target I build for I run into either EGL errors of GL errors.More info on this issue. It seems it happens with the profiles that requires to use an external runner (crucible, vk-gl-cts, deqp, ...). When executing, it tells it will run all the tests, but sometimes it just execute one test, other times 2, and other times none. It is in the last case when the error above is shown. Still don't know why.Found the problem in this commit: commit 9461d92301e72807eba4776a16a05207e3a16477 Author: Dylan Baker <[email protected]> Date: Mon Mar 26 15:23:17 2018 -0700 framework/profile: Add a __len__ method to TestProfileThis exposes a standard interface for getting the number of tests in aprofile, which is itself nice. It will also allow us to encapsulate the differences between the various profiles added in this series.Tested-by: Rafael Antognolli <[email protected]>I'm really having trouble reproducing this, the vulkan cts and crucible both run fine for me, no matter how many times I stop and start them. I even tried with python2 and couldn't reproduce. Can you give me some more information about your system?
I think I've hit this same issue on our CI.Symptoms match so that we sometimes run the whole 25k piglit gbm testset, sometimes we stop around the test 400-600. This behaviour can change with subsequent runs without rebooting the machine. Test where run is stopped is usually the same, and changes if filters change.
I can reproduce this with -d / --dry-run so the tests themselves are not an issue. Filtering with large -x / --exclude-tests might play a part. The command line is max 25kB, so there shouldn't be cutoff point with partial regex, which then would match too much.
I'm just starting to investigate where does the test list size drop so dramatically, probably by inserting testlist size debugs around to see where it takes me.
Environment: Ubuntu 18.04 LTS with default mesa Kernel: DRM-Tip HEAD or Ubuntu default.Commandline is built with bash array from blacklist. This looks correct, and sometimes works correctly. Eg
./piglit run tests/gpu ~/results -d -o -l verbose "${OPTIONS[@]}"
where $OPTIONS is an array of
'-x', 'timestamp-get',
'-x', 'glsl-routing', ...
Successful CI runlog:
http://gfx-ci.fi.intel.com/tree/drm-tip/CI_DRM_4148/pig-glk-j5005/run0.log
Unsuccessful CI runlog:
http://gfx-ci.fi.intel.com/tree/drm-tip/CI_DRM_4149/pig-glk-j5005/run0.log
Between those two runs, only kernel has changed.
The issue is easiest to reproduce with GLK. HSW seems to be somewhat
affected too, so the host speed might play a part.
Tomi -- Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
