https://bugs.kde.org/show_bug.cgi?id=492342
Bug ID: 492342
Summary: Nondeterministic results - Missing Nodes and Edges
across multiple of the same executables tested
Classification: Developer tools
Product: valgrind
Version: 3.18.1
Platform: Ubuntu
OS: Linux
Status: REPORTED
Severity: normal
Priority: NOR
Component: callgrind
Assignee: josef.weidendor...@gmx.de
Reporter: jo.al...@outlook.com
Target Milestone: ---
Created attachment 173064
--> https://bugs.kde.org/attachment.cgi?id=173064&action=edit
Refer to the "Observed Results" section for the explanation behind this.
Hi Valgrind Maintainers and Team!
I have been using Valgrind as a tool to check for nondeterministic behaviors
and I know that there are multiple subtools that Valgrind's suite comes with. I
would like to inquire about the following tools that exhibited flakiness as I
conducted these runs (you can read below regarding the methodology of how I
conducted such experiments). The terminology "projects" and "repositories" may
be used interchangeable here:
STEPS TO REPRODUCE
1. Use a GitHub Actions runner with Ubuntu 22.04 + Valgrind 3.18.1 pulled from
the Ubuntu package manager
2. Run your executable with Callgrind enabled.
3. Open Callgrind generated reports into KCachegrind to then export as graphViz
files.
OBSERVED RESULT
We tested 41 sample repositories against Valgrind and found that five of these
repositories seemed to report missing nodes and edges from Callgrind reports.
Here we have a sample repository that shows the differences in the missing
nodes + the edges. To explain why we show program A and B for missing nodes and
edges is because with program A, we have a large amount of missing edges that
would be difficult to show as a demonstration. In short, from the screenshots
(yellow represents the nodes being the same in each run):
Program A exhibited these nodes statistics (note that these clusters are part
of the Callgrind, but for demonstration purposes, we truncate to two):
- Run A produced (8 nodes in libc.so.6 & 13 libevent_core-2.2.so.1.0.0)
- Run B produced (3 nodes in libc.so.6 & 15 libevent_core-2.2.so.1.0.0)
Program B exhibited these edge statistics:
- Run A: 20 edges
- Run B: 32 edges
Another interesting point of observation is how in the missing nodes page, we
see that there is no overlap in nodes between two runs of the same program
despite keeping all executable configurations the same.
EXPECTED RESULT
Consistent nodes and edge graphs for a particular program, where we note that
36 repositories were able to yield the same or close number of nodes and edges
in a particular Callgrind report.
ADDITIONAL INFORMATION
Refer to the following attachments.
--
You are receiving this mail because:
You are watching all bug changes.
