From: Pekka Paalanen <[email protected]> Hi,
I and Ben have been fighting with benchmarking on the Raspberry Pi 1 for a long time. The problem is unexpected performance differences. Insignificant changes to the code can cause significant changes to lowlevel-blt-bench performance results. The change can be a speed up or a slow down, and it can happen on portions of the code we never modified. Over the course, we have developed our own benchmarking scripts, that execute lowlevel-blt-bench many times and then do a statistical analysis (Student's t-test) on the significance of the change. When I say "significant changes", I mean in the t-test sense. I have stabilized the running environment as far as I could: RPi 1 is using an usb-disk, swap is not on SD-card, the script 'sync's the disks between each execution, init level is changed to 1 to eliminate as much of background processes as possible, the network is unplugged, and the monitor is forced to stay on (scanout consumes memory bandwidth). None of this really helped with the unexpected differences - they are not random, but very repeatable *if* one takes great care to just iterate on the same program binaries. We also ruled out drifting CPU clock speeds. The scripts run each code revision interleaved; each iteration consist of running every revision once. I have done 30 - 100 iterations, the total runtime being usually somewhere between 4 and 20 hours, depending. Any drift in clock speed or such would be removed in the statistical analysis. We also asked the hardware people, and such drift should just not happen. CPU scaling should not be happening either on RPi1. The magnitude of the unexpected changes is sometimes fairly big, in the order of 5%, but we have seen anomalies even up to 20%. A thing that explains a great deal of these anomalies, but not all of it, has something to do with function addresses. There are hypotheses that it might have to do with the branch predictor and its cache. We made a test targeting exactly that idea: pick a fast path function that seems to be most susceptible to unexpected changes, pad it with x nops before the function start and N-x nops after the function end. We never execute those nops, but changing x changes the function start address while keeping everything else in the whole binary in the same place. The results were mind-boggling: depending on the function starting address, the src_8888_8888 L1 test of lowlevel-blt-bench went either 355 Mpx/s or 470 Mpx/s. There does not seem to be any predictable pattern on which addresses are "fast" and which are "slow". Obviously this will screw up our benchmarks, because a change in an unrelated function may cause another function's address to shift, and therefore change its performance. See [1] for the plot. How should we benchmark on Raspberry Pi 1 then? When an arbitrary application loads libpixman-1, we have no guarantees about the loading addresses, even if we could make the Pixman build "stable" in that respect. We ended up with two major points: - A patch adding a new fast path needs to be split in two: the first patch adds all the code, but the code is disabled. The second patch enables the new code without changing any function addresses. This should eliminate most of the unexpected performance differences. - To get an average of what performance applications might be seeing, we must make sure the libpixman-1 loading address is randomized, and execute the benchmark program many times. This can be achieved by letting Pixman be a shared library that the benchmark, e.g. lowlevel-blt-bench, links dynamically, and ensuring ASLR does the randomization. Of course, it's always good to minimize any possible interference, so I will stick to my "stabilized" running environment. With these insights, here are the patches for two new fast paths: over_n_8888 as both a generic C version, and an armv6 assembly version. The patches that enable fast paths contain the essential benchmark results. At the end of this email, you can find the complete performance comparisons not only for the patches that enable fast paths but also for the patches that are not supposed to affect performance. In the future I do not intend to include these reports. The benchmarks were run with 30 iterations (executions of lowlevel-blt-bench per code revision). References: The statistical analysis script https://github.com/bavison/perfcmp My test scripts and data https://git.collabora.com/cgit/user/pq/pixman-benchmarking.git/ [1] The plot of alignment vs. performance https://git.collabora.com/cgit/user/pq/pixman-benchmarking.git/plain/octave/figures/fig-src-8888-8888-L1.pdf The Pixman source used to run the alignment vs. performance test: https://git.collabora.com/cgit/user/pq/pixman.git/log/?h=interbench-20150702 Thanks, pq Ben Avison (2): pixman-fast-path: Add over_n_8888 fast path (disabled) armv6: Add over_n_8888 fast path (disabled) Pekka Paalanen (2): pixman-fast-path: enable over_n_8888 armv6: enable over_n_8888 pixman/pixman-arm-simd-asm.S | 41 +++++++++++++++++++++++++++++++++++++++++ pixman/pixman-arm-simd.c | 6 ++++++ pixman/pixman-fast-path.c | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+) -- 2.4.6 Benchmark results ****************** Before: upstream master, no patches applied After: patch 1 Expectation: no performance changes Test of feature 'in_8888_8' Before After Mean StdDev Mean StdDev Confidence Change L1 13.2 0.03 13.1 0.03 100.00% -0.6% L2 9.5 0.16 9.5 0.19 35.51% -0.2% (insignificant) M 9.7 0.00 9.7 0.01 17.39% +0.0% (insignificant) HT 7.8 0.02 7.8 0.02 92.52% +0.1% (insignificant) VT 7.7 0.02 7.7 0.02 99.74% +0.2% R 7.3 0.01 7.3 0.01 100.00% +0.2% RT 4.0 0.04 4.1 0.05 100.00% +1.5% At most 4 outliers rejected per test per set. Test of feature 'in_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 19.1 0.06 19.2 0.07 100.00% +0.4% L2 15.3 0.44 15.5 0.50 83.84% +1.1% (insignificant) M 12.8 0.00 12.8 0.00 71.74% +0.0% (insignificant) HT 11.3 0.03 11.3 0.03 99.97% +0.2% VT 11.0 0.02 11.0 0.03 99.81% +0.2% R 10.7 0.02 10.7 0.02 99.85% +0.2% RT 6.5 0.08 6.6 0.09 92.59% +0.6% (insignificant) At most 0 outliers rejected per test per set. Test of feature 'in_reverse_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 32.2 0.06 32.3 0.09 100.00% +0.6% L2 17.8 0.61 17.9 0.59 63.65% +0.8% (insignificant) M 16.9 0.02 16.9 0.02 50.08% -0.0% (insignificant) HT 12.1 0.03 12.0 0.03 83.85% -0.1% (insignificant) VT 11.8 0.04 11.8 0.03 71.18% +0.1% (insignificant) R 11.3 0.03 11.3 0.02 97.65% -0.2% (insignificant) RT 6.1 0.11 6.1 0.07 85.29% -0.6% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 37.8 0.12 38.0 0.07 100.00% +0.7% L2 27.5 0.74 27.7 0.59 65.53% +0.6% (insignificant) M 26.9 0.03 26.9 0.03 31.38% -0.0% (insignificant) HT 14.9 0.04 14.9 0.05 40.27% -0.0% (insignificant) VT 14.1 0.04 14.0 0.04 99.37% -0.2% R 15.1 0.05 15.1 0.06 7.05% -0.0% (insignificant) RT 7.3 0.13 7.3 0.14 7.64% +0.0% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_8888_8_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.3 0.01 5.3 0.01 100.00% +0.3% L2 4.6 0.04 4.6 0.04 36.58% -0.1% (insignificant) M 4.5 0.00 4.5 0.00 43.81% +0.0% (insignificant) HT 3.9 0.01 3.9 0.01 67.48% +0.0% (insignificant) VT 3.9 0.01 3.9 0.00 91.95% +0.1% (insignificant) R 3.8 0.01 3.8 0.01 94.80% -0.1% (insignificant) RT 2.3 0.01 2.3 0.02 55.99% -0.1% (insignificant) At most 5 outliers rejected per test per set. Test of feature 'over_8888_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.7 0.01 5.7 0.01 66.51% +0.0% (insignificant) L2 5.0 0.03 5.0 0.05 35.72% +0.1% (insignificant) M 4.8 0.00 4.8 0.00 99.95% +0.0% HT 4.5 0.01 4.5 0.01 100.00% +0.2% VT 4.5 0.01 4.5 0.01 100.00% +0.2% R 4.3 0.01 4.4 0.01 100.00% +0.2% RT 2.9 0.02 2.9 0.03 55.17% +0.2% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 8.3 0.02 8.4 0.03 82.45% +0.1% (insignificant) L2 8.1 0.02 8.1 0.01 72.48% -0.1% (insignificant) M 7.4 0.00 7.4 0.00 96.90% +0.0% (insignificant) HT 7.0 0.01 7.0 0.01 99.71% +0.2% VT 6.9 0.01 7.0 0.01 99.94% +0.2% R 6.8 0.01 6.8 0.01 43.44% +0.0% (insignificant) RT 4.6 0.05 4.5 0.05 90.51% -0.5% (insignificant) At most 5 outliers rejected per test per set. Test of feature 'over_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 12.4 0.04 12.5 0.03 100.00% +0.6% L2 11.1 0.02 11.1 0.02 47.25% -0.0% (insignificant) M 9.4 0.00 9.4 0.00 95.77% +0.0% (insignificant) HT 8.5 0.02 8.5 0.02 74.90% +0.1% (insignificant) VT 8.4 0.02 8.4 0.02 64.04% +0.0% (insignificant) R 8.2 0.02 8.2 0.01 98.26% +0.1% (insignificant) RT 5.5 0.08 5.5 0.05 11.15% +0.0% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'src_0565_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 480.8 4.67 514.0 6.57 100.00% +6.9% L2 84.1 4.83 82.1 3.41 93.68% -2.4% (insignificant) M 128.2 0.15 128.2 0.14 38.20% +0.0% (insignificant) HT 49.7 0.46 49.9 0.37 92.51% +0.4% (insignificant) VT 44.1 0.34 44.2 0.29 87.47% +0.3% (insignificant) R 39.1 0.32 39.2 0.33 75.09% +0.2% (insignificant) RT 12.4 0.34 12.5 0.31 75.39% +0.8% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'src_1555_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 25.0 0.10 24.1 0.13 100.00% -3.5% L2 19.0 0.26 18.9 0.39 87.17% -0.7% (insignificant) M 20.0 0.02 20.0 0.02 18.97% +0.0% (insignificant) HT 12.5 0.04 12.4 0.06 100.00% -0.9% VT 12.4 0.05 12.3 0.06 100.00% -0.8% R 11.8 0.05 11.7 0.05 100.00% -0.9% RT 5.4 0.11 5.4 0.09 99.82% -1.5% At most 1 outliers rejected per test per set. Test of feature 'src_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 468.1 4.43 478.2 9.26 100.00% +2.2% L2 58.0 4.34 58.5 4.14 31.74% +0.8% (insignificant) M 87.4 0.13 87.4 0.13 24.42% -0.0% (insignificant) HT 36.5 0.15 36.7 0.22 99.80% +0.4% VT 32.7 0.17 32.8 0.19 76.34% +0.2% (insignificant) R 31.0 0.14 31.2 0.19 99.97% +0.6% RT 11.5 0.18 11.7 0.23 100.00% +2.2% At most 1 outliers rejected per test per set. ****************** Before: patch 1 After: patch 1+2 Expectation: performance improvement on over_n_8888 Test of feature 'in_8888_8' Before After Mean StdDev Mean StdDev Confidence Change L1 13.1 0.03 13.1 0.03 99.09% -0.2% L2 9.5 0.19 9.5 0.14 2.46% +0.0% (insignificant) M 9.7 0.01 9.6 0.01 98.57% -0.1% (insignificant) HT 7.8 0.02 7.8 0.02 17.57% +0.0% (insignificant) VT 7.7 0.02 7.7 0.02 1.25% +0.0% (insignificant) R 7.3 0.01 7.3 0.01 99.41% -0.1% RT 4.1 0.05 4.0 0.04 100.00% -1.3% At most 2 outliers rejected per test per set. Test of feature 'in_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 19.2 0.07 19.2 0.08 67.38% +0.1% (insignificant) L2 15.5 0.50 15.4 0.52 47.56% -0.5% (insignificant) M 12.8 0.00 12.8 0.00 43.93% -0.0% (insignificant) HT 11.3 0.03 11.3 0.03 10.26% +0.0% (insignificant) VT 11.0 0.03 11.0 0.03 53.19% +0.0% (insignificant) R 10.7 0.02 10.7 0.03 13.89% -0.0% (insignificant) RT 6.6 0.09 6.6 0.09 16.12% -0.1% (insignificant) At most 0 outliers rejected per test per set. Test of feature 'in_reverse_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 32.3 0.09 32.3 0.07 1.81% -0.0% (insignificant) L2 17.9 0.59 17.9 0.62 6.42% +0.1% (insignificant) M 16.9 0.02 16.9 0.02 48.34% +0.0% (insignificant) HT 12.0 0.03 12.1 0.04 97.10% +0.2% (insignificant) VT 11.8 0.03 11.8 0.04 97.84% +0.2% (insignificant) R 11.3 0.02 11.3 0.03 91.88% +0.1% (insignificant) RT 6.1 0.07 6.1 0.10 96.81% +0.8% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 38.0 0.07 38.0 0.13 34.92% +0.0% (insignificant) L2 27.7 0.59 27.7 0.61 9.76% +0.1% (insignificant) M 26.9 0.03 26.9 0.03 15.34% +0.0% (insignificant) HT 14.9 0.05 14.9 0.05 79.64% +0.1% (insignificant) VT 14.0 0.04 14.1 0.04 96.23% +0.2% (insignificant) R 15.1 0.06 15.1 0.06 31.84% +0.0% (insignificant) RT 7.3 0.14 7.3 0.14 12.11% -0.1% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_8888_8_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.3 0.01 5.3 0.01 60.93% -0.0% (insignificant) L2 4.6 0.04 4.6 0.04 60.95% +0.2% (insignificant) M 4.5 0.00 4.5 0.00 68.24% -0.0% (insignificant) HT 3.9 0.01 3.9 0.01 62.31% +0.0% (insignificant) VT 3.9 0.00 3.9 0.01 61.17% +0.0% (insignificant) R 3.8 0.01 3.8 0.00 87.38% +0.1% (insignificant) RT 2.3 0.02 2.3 0.01 35.89% -0.1% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_8888_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.7 0.01 5.7 0.01 35.37% +0.0% (insignificant) L2 5.0 0.05 5.0 0.04 55.24% +0.2% (insignificant) M 4.8 0.00 4.8 0.00 94.55% -0.0% (insignificant) HT 4.5 0.01 4.5 0.01 50.62% +0.0% (insignificant) VT 4.5 0.01 4.5 0.01 80.60% +0.1% (insignificant) R 4.4 0.01 4.4 0.01 65.97% +0.0% (insignificant) RT 2.9 0.03 2.9 0.02 49.54% +0.2% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 8.4 0.03 8.4 0.03 20.63% +0.0% (insignificant) L2 8.1 0.01 8.1 0.01 19.00% +0.0% (insignificant) M 7.4 0.00 7.4 0.00 7.43% -0.0% (insignificant) HT 7.0 0.01 7.0 0.01 33.74% +0.0% (insignificant) VT 7.0 0.01 7.0 0.01 46.73% +0.0% (insignificant) R 6.8 0.01 6.8 0.01 53.98% +0.0% (insignificant) RT 4.5 0.05 4.5 0.04 14.83% +0.1% (insignificant) At most 5 outliers rejected per test per set. Test of feature 'over_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 12.5 0.03 21.2 0.05 100.00% +69.5% L2 11.1 0.02 17.4 0.01 100.00% +57.3% M 9.4 0.00 13.6 0.00 100.00% +45.1% HT 8.5 0.02 12.2 0.02 100.00% +43.0% VT 8.4 0.02 11.9 0.02 100.00% +41.7% R 8.2 0.01 11.5 0.02 100.00% +40.5% RT 5.5 0.05 7.6 0.08 100.00% +39.1% At most 2 outliers rejected per test per set. Test of feature 'src_0565_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 514.0 6.57 507.0 5.16 99.99% -1.4% L2 82.1 3.41 83.1 4.83 66.63% +1.3% (insignificant) M 128.2 0.14 128.2 0.13 47.76% -0.0% (insignificant) HT 49.9 0.37 49.7 0.36 95.44% -0.4% (insignificant) VT 44.2 0.29 44.2 0.26 0.41% -0.0% (insignificant) R 39.2 0.33 39.1 0.29 63.60% -0.2% (insignificant) RT 12.5 0.31 12.4 0.26 67.37% -0.6% (insignificant) At most 6 outliers rejected per test per set. Test of feature 'src_1555_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 24.1 0.13 24.2 0.13 61.78% +0.1% (insignificant) L2 18.9 0.39 18.7 0.19 96.50% -0.9% (insignificant) M 20.0 0.02 20.0 0.03 60.83% -0.0% (insignificant) HT 12.4 0.06 12.4 0.05 93.43% +0.2% (insignificant) VT 12.3 0.06 12.3 0.06 77.64% +0.2% (insignificant) R 11.7 0.05 11.7 0.05 29.04% +0.0% (insignificant) RT 5.4 0.09 5.4 0.07 83.90% +0.6% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'src_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 478.2 9.26 479.5 5.48 46.79% +0.3% (insignificant) L2 58.5 4.14 58.2 4.11 21.77% -0.5% (insignificant) M 87.4 0.13 87.4 0.12 62.93% +0.0% (insignificant) HT 36.7 0.22 36.7 0.20 73.41% +0.2% (insignificant) VT 32.8 0.19 32.8 0.21 35.67% +0.1% (insignificant) R 31.2 0.19 31.2 0.18 68.76% +0.2% (insignificant) RT 11.7 0.23 11.8 0.26 89.51% +0.9% (insignificant) At most 2 outliers rejected per test per set. ****************** Before: patch 1+2 After: patch 1+2+3 Expectation: no performance changes Test of feature 'in_8888_8' Before After Mean StdDev Mean StdDev Confidence Change L1 13.1 0.03 13.3 0.03 100.00% +1.9% L2 9.5 0.14 9.5 0.17 20.38% +0.1% (insignificant) M 9.6 0.01 9.7 0.00 99.52% +0.1% HT 7.8 0.02 7.8 0.01 97.10% -0.1% (insignificant) VT 7.7 0.02 7.7 0.01 99.05% -0.1% R 7.3 0.01 7.3 0.01 2.45% -0.0% (insignificant) RT 4.0 0.04 4.0 0.05 1.26% -0.0% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'in_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 19.2 0.08 19.1 0.06 100.00% -0.6% L2 15.4 0.52 15.4 0.45 38.75% -0.4% (insignificant) M 12.8 0.00 12.8 0.00 98.05% +0.0% (insignificant) HT 11.3 0.03 11.3 0.03 99.99% -0.3% VT 11.0 0.03 11.0 0.03 99.23% -0.2% R 10.7 0.03 10.7 0.02 97.90% -0.1% (insignificant) RT 6.6 0.09 6.6 0.09 7.24% +0.0% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'in_reverse_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 32.3 0.07 32.4 0.09 79.24% +0.1% (insignificant) L2 17.9 0.62 17.9 0.48 18.75% -0.2% (insignificant) M 16.9 0.02 16.9 0.02 49.64% +0.0% (insignificant) HT 12.1 0.04 12.1 0.03 71.54% -0.1% (insignificant) VT 11.8 0.04 11.8 0.02 99.96% -0.3% R 11.3 0.03 11.3 0.02 22.90% -0.0% (insignificant) RT 6.1 0.10 6.1 0.07 77.15% -0.4% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 38.0 0.13 38.0 0.14 8.75% -0.0% (insignificant) L2 27.7 0.61 27.5 0.83 73.75% -0.8% (insignificant) M 26.9 0.03 26.9 0.03 3.72% -0.0% (insignificant) HT 14.9 0.05 14.9 0.06 15.08% -0.0% (insignificant) VT 14.1 0.04 14.0 0.04 99.80% -0.3% R 15.1 0.06 15.1 0.06 46.46% -0.1% (insignificant) RT 7.3 0.14 7.3 0.13 15.09% +0.1% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_8888_8_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.3 0.01 5.3 0.01 100.00% -0.2% L2 4.6 0.04 4.6 0.04 21.09% -0.1% (insignificant) M 4.5 0.00 4.5 0.00 60.21% +0.0% (insignificant) HT 3.9 0.01 3.9 0.01 0.83% -0.0% (insignificant) VT 3.9 0.01 3.9 0.01 30.43% +0.0% (insignificant) R 3.8 0.00 3.8 0.01 94.02% +0.1% (insignificant) RT 2.3 0.01 2.3 0.02 99.87% +0.6% At most 1 outliers rejected per test per set. Test of feature 'over_8888_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.7 0.01 5.7 0.01 81.08% +0.1% (insignificant) L2 5.0 0.04 5.0 0.04 64.46% -0.2% (insignificant) M 4.8 0.00 4.8 0.00 6.04% -0.0% (insignificant) HT 4.5 0.01 4.5 0.01 100.00% -0.2% VT 4.5 0.01 4.5 0.01 100.00% -0.2% R 4.4 0.01 4.4 0.01 97.94% -0.1% (insignificant) RT 2.9 0.02 2.9 0.02 99.98% -0.9% At most 1 outliers rejected per test per set. Test of feature 'over_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 8.4 0.03 8.4 0.02 31.01% +0.0% (insignificant) L2 8.1 0.01 8.1 0.01 14.41% +0.0% (insignificant) M 7.4 0.00 7.4 0.00 22.23% +0.0% (insignificant) HT 7.0 0.01 7.0 0.01 97.96% -0.1% (insignificant) VT 7.0 0.01 6.9 0.01 96.55% -0.1% (insignificant) R 6.8 0.01 6.8 0.01 93.76% -0.1% (insignificant) RT 4.5 0.04 4.6 0.05 94.47% +0.5% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'over_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 21.2 0.05 21.2 0.05 41.70% +0.0% (insignificant) L2 17.4 0.01 17.4 0.01 100.00% -0.1% M 13.6 0.00 13.6 0.00 100.00% -0.0% HT 12.2 0.02 12.2 0.04 73.85% -0.1% (insignificant) VT 11.9 0.02 11.8 0.03 44.38% -0.0% (insignificant) R 11.5 0.02 11.5 0.03 96.20% -0.1% (insignificant) RT 7.6 0.08 7.6 0.11 62.74% -0.3% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'src_0565_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 507.0 5.16 478.3 8.23 100.00% -5.7% L2 83.1 4.83 82.6 4.39 32.21% -0.6% (insignificant) M 128.2 0.13 128.3 0.13 76.78% +0.0% (insignificant) HT 49.7 0.36 49.6 0.34 76.47% -0.2% (insignificant) VT 44.2 0.26 43.9 0.18 100.00% -0.8% R 39.1 0.29 39.1 0.30 4.50% -0.0% (insignificant) RT 12.4 0.26 12.3 0.28 69.77% -0.6% (insignificant) At most 6 outliers rejected per test per set. Test of feature 'src_1555_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 24.2 0.13 25.4 0.12 100.00% +5.1% L2 18.7 0.19 18.9 0.40 99.19% +1.2% M 20.0 0.03 20.0 0.02 96.94% +0.1% (insignificant) HT 12.4 0.05 12.4 0.04 96.56% +0.2% (insignificant) VT 12.3 0.06 12.3 0.05 84.07% +0.2% (insignificant) R 11.7 0.05 11.7 0.05 99.71% +0.3% RT 5.4 0.07 5.4 0.08 7.45% +0.0% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'src_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 479.5 5.48 476.8 7.58 85.00% -0.6% (insignificant) L2 58.2 4.11 58.7 4.27 39.42% +1.0% (insignificant) M 87.4 0.12 87.4 0.13 15.61% -0.0% (insignificant) HT 36.7 0.20 36.7 0.23 69.17% -0.2% (insignificant) VT 32.8 0.21 32.7 0.22 98.61% -0.4% (insignificant) R 31.2 0.18 31.2 0.16 77.16% -0.2% (insignificant) RT 11.8 0.26 11.7 0.13 99.78% -1.4% At most 4 outliers rejected per test per set. ****************** Before: patch 1+2+3 After: patch 1+2+3+4 Expectation: performance improvement on over_n_8888 Test of feature 'in_8888_8' Before After Mean StdDev Mean StdDev Confidence Change L1 13.3 0.03 13.3 0.05 99.85% -0.2% L2 9.5 0.17 9.4 0.14 84.59% -0.6% (insignificant) M 9.7 0.00 9.7 0.00 12.02% -0.0% (insignificant) HT 7.8 0.01 7.8 0.02 29.74% -0.0% (insignificant) VT 7.7 0.01 7.7 0.02 24.23% -0.0% (insignificant) R 7.3 0.01 7.3 0.02 44.23% +0.0% (insignificant) RT 4.0 0.05 4.0 0.05 79.26% -0.4% (insignificant) At most 4 outliers rejected per test per set. Test of feature 'in_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 19.1 0.06 19.1 0.07 57.49% +0.1% (insignificant) L2 15.4 0.45 15.5 0.51 61.13% +0.7% (insignificant) M 12.8 0.00 12.8 0.00 31.36% -0.0% (insignificant) HT 11.3 0.03 11.3 0.03 36.66% +0.0% (insignificant) VT 11.0 0.03 11.0 0.03 49.05% +0.0% (insignificant) R 10.7 0.02 10.7 0.03 36.66% +0.0% (insignificant) RT 6.6 0.09 6.6 0.11 31.99% +0.2% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'in_reverse_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 32.4 0.09 32.4 0.10 80.57% +0.1% (insignificant) L2 17.9 0.48 17.9 0.44 32.45% +0.3% (insignificant) M 16.9 0.02 17.0 0.02 18.84% +0.0% (insignificant) HT 12.1 0.03 12.0 0.03 49.37% -0.0% (insignificant) VT 11.8 0.02 11.8 0.03 5.08% +0.0% (insignificant) R 11.3 0.02 11.3 0.02 56.36% +0.0% (insignificant) RT 6.1 0.07 6.1 0.10 44.32% +0.2% (insignificant) At most 2 outliers rejected per test per set. Test of feature 'over_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 38.0 0.14 38.1 0.12 49.85% +0.1% (insignificant) L2 27.5 0.83 27.6 0.83 34.98% +0.4% (insignificant) M 26.9 0.03 26.9 0.03 23.94% +0.0% (insignificant) HT 14.9 0.06 14.9 0.06 18.90% -0.0% (insignificant) VT 14.0 0.04 14.0 0.06 10.01% -0.0% (insignificant) R 15.1 0.06 15.1 0.07 16.20% +0.0% (insignificant) RT 7.3 0.13 7.3 0.17 40.59% +0.3% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'over_8888_8_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.3 0.01 5.3 0.01 47.95% -0.0% (insignificant) L2 4.6 0.04 4.6 0.03 35.24% -0.1% (insignificant) M 4.5 0.00 4.5 0.00 5.89% -0.0% (insignificant) HT 3.9 0.01 3.9 0.01 42.81% -0.0% (insignificant) VT 3.9 0.01 3.9 0.01 80.10% -0.1% (insignificant) R 3.8 0.01 3.8 0.01 68.22% -0.0% (insignificant) RT 2.3 0.02 2.3 0.02 55.04% -0.2% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_8888_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 5.7 0.01 5.7 0.01 91.56% -0.1% (insignificant) L2 5.0 0.04 5.0 0.03 88.04% +0.3% (insignificant) M 4.8 0.00 4.8 0.00 89.77% +0.0% (insignificant) HT 4.5 0.01 4.5 0.01 87.14% +0.1% (insignificant) VT 4.5 0.01 4.5 0.01 91.46% +0.1% (insignificant) R 4.4 0.01 4.4 0.01 92.73% +0.1% (insignificant) RT 2.9 0.02 2.9 0.03 4.10% -0.0% (insignificant) At most 1 outliers rejected per test per set. Test of feature 'over_n_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 8.4 0.02 8.3 0.03 98.07% -0.2% (insignificant) L2 8.1 0.01 8.1 0.01 97.97% +0.1% (insignificant) M 7.4 0.00 7.4 0.00 27.60% +0.0% (insignificant) HT 7.0 0.01 7.0 0.01 52.87% -0.0% (insignificant) VT 6.9 0.01 6.9 0.01 1.96% +0.0% (insignificant) R 6.8 0.01 6.8 0.01 71.27% -0.0% (insignificant) RT 4.6 0.05 4.5 0.04 100.00% -1.2% At most 4 outliers rejected per test per set. Test of feature 'over_n_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 21.2 0.05 45.4 0.15 100.00% +113.8% L2 17.4 0.01 43.2 0.03 100.00% +148.0% M 13.6 0.00 42.4 0.02 100.00% +211.4% HT 12.2 0.04 25.4 0.13 100.00% +109.1% VT 11.8 0.03 22.3 0.09 100.00% +88.5% R 11.5 0.03 23.2 0.10 100.00% +102.2% RT 7.6 0.11 11.5 0.19 100.00% +51.2% At most 1 outliers rejected per test per set. Test of feature 'src_0565_0565' Before After Mean StdDev Mean StdDev Confidence Change L1 478.3 8.23 479.2 12.82 23.88% +0.2% (insignificant) L2 82.6 4.39 82.8 4.44 10.21% +0.2% (insignificant) M 128.3 0.13 128.2 0.14 55.57% -0.0% (insignificant) HT 49.6 0.34 49.7 0.27 81.52% +0.2% (insignificant) VT 43.9 0.18 44.1 0.26 99.34% +0.4% R 39.1 0.30 39.1 0.31 1.40% -0.0% (insignificant) RT 12.3 0.28 12.4 0.27 40.16% +0.3% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'src_1555_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 25.4 0.12 25.4 0.13 10.64% +0.0% (insignificant) L2 18.9 0.40 19.0 0.32 15.56% +0.1% (insignificant) M 20.0 0.02 20.0 0.02 18.25% -0.0% (insignificant) HT 12.4 0.04 12.4 0.03 6.74% -0.0% (insignificant) VT 12.3 0.05 12.3 0.04 55.63% +0.1% (insignificant) R 11.7 0.05 11.7 0.03 47.13% +0.1% (insignificant) RT 5.4 0.08 5.4 0.07 14.83% -0.1% (insignificant) At most 3 outliers rejected per test per set. Test of feature 'src_8888_8888' Before After Mean StdDev Mean StdDev Confidence Change L1 476.8 7.58 477.9 5.47 45.12% +0.2% (insignificant) L2 58.7 4.27 57.6 4.91 65.35% -1.9% (insignificant) M 87.4 0.13 87.4 0.12 58.01% -0.0% (insignificant) HT 36.7 0.23 36.6 0.16 63.26% -0.1% (insignificant) VT 32.7 0.22 32.7 0.16 53.57% -0.1% (insignificant) R 31.2 0.16 31.1 0.15 78.92% -0.2% (insignificant) RT 11.7 0.13 11.7 0.22 31.55% +0.2% (insignificant) At most 6 outliers rejected per test per set. _______________________________________________ Pixman mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/pixman
