fuzzcheck-rs

2022-08-21 ยท 2 min read

Coverage-guided, in-process, structure-aware, mutation-based fuzzing without the intermediate raw bytes serde step.

potential improvements #

  • from a maintainence standpoint, it's a pain to support both proptest and fuzzcheck-rs simultaneously.
    • can fuzzcheck-rs completely subsume the use cases where we normally use proptest?
    • as normal, I can run cargo fuzzcheck <my-fuzz-test> for continuous fuzzing.
    • however, I should be able to hit cargo test and all my fuzzers run in a "simplified" checking/testing mode where:
      • there is a default max duration (5 sec?) / max test cases value (10,000 execs?).
      • generated corpus values/artifacts aren't saved to disk (unless something fails).
      • IF a failing test case is found, either:
        1. immediately start minifying the test input for some duration (15 sec?).
        2. print out a copy-pasteable command the will minify that specific test.
      • If coverage is too expensive to track during tests, it wouldn't even be that bad to just disable it when cargo test'ing.
        • in fact, I'm not sure how coverage would work if the tests are run in the same process.
        • a #[fuzzcheck::test] annotation could mandate tests use something like rusty-fork to fork the test process before running.
    • One advantage of proptest is that you can easily generate complex test inputs by composing the generator combinators.
      • in contrast, it wasn't clear how to build a custom mutator composed of submutators (from my very brief look).
    • By using the same fuzzers for both continuous fuzzing and unit testing, we prevent fuzzers from going stale or otherwise drifting from the implementation.
      • This was a very real problem in a previous project. The continous fuzzers ran out-of-band from the CI-blocking test suite, meaning PRs weren't blocked when they accidentally broke a fuzzer. People often forgot to add a #[test] that ran the fuzz test on a few inputs.
    • People seemed to adopt proptest when writing tests, but getting people to write fuzzers was a challenge.
  • is there some way to avoid spamming the #[no_coverage] attribute everywhere?
  • can we simplify the on-boarding process and reduce the amount of #[cfg(fuzzing)] needed?