I have some additional comments though I don't work on TVM/Relay; I just helped Steven out with some parts of the fuzzer.
[quote="slyubomirsky, post:1, topic:12234"] Fuzzing infrastructure poses the most questions: Should it be defined within TVM’s codebase, or live separately? Where should the scripts for invoking it be defined as well? [/quote] I'm in favor of just keeping the fuzzer (the main driving part of the fuzzer) in the top level repo since it is pretty small, but I don't think it'd be much of a difference keeping it in another repo. As to where to storge component-specific fuzzing parts like oracles or pass-specific fuzzing harnesses, I think the two best options are: 1. Keep these things next to the code. For example, for transformation-specific fuzzing wrappers, just put these wrappers next to their transforms in the `src/relay/transforms`. 1. Keep these things in some tooling or test directory. For example, those same transformation-specific fuzzing wrappers would instead go in `tests/python/relay` (or some subdir there). I'm not super familiar with Python, but I think that with the way Python modules work and the fuzzer currently written in Python, maybe the second option would be easier (?). Probably the most important part of the infrastructure to figure out is how this is going to run. Since it sounds like the fuzzer will run quite frequently, the runs should probably be automated by some CI-fuzzing tool that has statistic and bug reporting abilities. I don't have much experience with CI-based fuzzing toolchains, but I would like to throw ClusterFuzzLite (https://github.com/google/clusterfuzzlite) into the mix for consideration. It is a simplified version of Google's ClusterFuzz, runs in CI, has the reporting facilities, and can handle mixed Python/C++ projects. I found this a ujson fuzzer for ClusterFuzzLite: https://github.com/google/oss-fuzz/blob/master/projects/ujson/hypothesis_structured_fuzzer.py which mixes Python and C; this could be used as a starting point or template to getting the Relay fuzzer up and running. [quote="slyubomirsky, post:1, topic:12234"] I would be interested in technical advice on tracking code coverage, especially across the C++ and Python boundary, if it is desirable (I think it is). [/quote] For an earlier evaluation of the fuzzer, we used llvm's SourceBasedCodeCoverage and SanitizerCoverage to instrument and track code coverage of the C++ parts of Relay. This injects code coverage tracking code at compile time, so we couldn't get full coverage data because of the Python/C++ split in the fuzzer, but unless others have better ways of getting coverage data, I'm guessing this will be the way it gets coverage to start. --- [Visit Topic](https://discuss.tvm.apache.org/t/rfc-type-directed-relay-fuzzing-library/12234/2) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/3e54be0404be07bf97007b9069f5527c589cda6b641da89405d9c4b892362f3b).
