rr with rust

2022-07-30 ยท 4 min read

github: https://github.com/rr-debugger/rr site: https://rr-project.org/

what #

  • rr is a lightweight tool for recording, replaying, and debugging applications.
  • rr records deterministic traces of the entire program execution (including all child processes and spawned threads), which can be replayed many times or even run in reverse.
  • rr models a single-threaded machine. It creates a deterministic trace by intercepting and recording all Linux syscalls for later replay. rr also controls the thread and process scheduling so we can even determinstically replay concurrent programs.

why #

  • you have a flaky concurrency bug that only manifests occasionally.
    • ==> use rr to record 1000s of execution traces (also try --chaos mode!) until a lucky one finally triggers the bug. Later, replay the failing trace as many times as you please.
  • if you're like me, you're terrible at setting breakpoints.
    • ==> with rr you can just replay the same exact execution with better breakpoints; no state changing under your feet between runs.
  • you have a backtrace but the offending code isn't in any of the backtrace frames
    • ==> just execute the program backwards until you hit the problem.
  • your fuzzer sometimes finds a bug after 10 hr, but when you run again the bug ain't there
    • ==> run your fuzzers inside rr. when the fuzzer hits a bug, you can easily replay it.

Better yet, use rr as a general purpose gdb enhancement : )

requirements #

  • Linux kernel >= 3.11
    • Works inside a VM if the VM supports virtualization of hardware perf counters. (incl. VMware and KVM; does not incl. Xen).
  • One of:

setup # 

$ sudo apt install rr

You will also need gdb installed, since rr replay is "just" a wrapper around gdb.

$ sudo apt install gdb

When debugging rust code, there are a few extra command flags that rr needs. Tests are also slightly more annoying to debug, since cargo runs your test binary in a sub-process. The cargo rr tool makes the debugging process painless.

$ cargo install cargo-rr

record a trace for deterministic replay # 

# Record a test
$ cargo rr test my_failing_test

# Record a binary
$ cargo rr run --bin=foo -- --arg=bar

NOTE: if you get an error message like rr needs /proc/sys/kernel/perf_event_paranoid <= 1, but it is 2. then try:

$ echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid

replay last trace # 

$ cargo rr replay

If you prefer a more visual debugging experience and hate the letter l, try the TUI debugger:

(rr) <CTRL-X a>

# if that doesn't work, try
(rr) tui enable

See: gdb ref > TUI keybindings not working

debugging #

Debugging with rr looks the same as debugging with gdb, except you have a few extra commands for executing backwards.

(rr) help reverse-continue
reverse-continue, rc
Continue program being debugged but run it in reverse.
If proceeding from breakpoint, a number N may be used as an argument,
which means to set the ignore count of that breakpoint to N - 1 (so that
the breakpoint won't break until the Nth time it is reached).

(rr) help reverse-step
reverse-step, rs
Step program backward until it reaches the beginning of another source line.
Argument N means do this N times (or till program stops for another reason).

(rr) help reverse-next
reverse-next, rn
Step program backward, proceeding through subroutine calls.
Like the "reverse-step" command as long as subroutine calls do not happen;
when they do, the call is treated as one instruction.
Argument N means do this N times (or till program stops for another reason).

...

(rr) help r<TAB>
rbreak                    return                    rr-history-push
rc                        reverse-continue          rr-hook-run
record                    reverse-finish            rr-set-suppress-run-hook
refresh                   reverse-next              rr-where
remote                    reverse-nexti             rsi
remove-inferiors          reverse-search            run
remove-symbol-file        reverse-step              running
restart                   reverse-stepi             rwatch
restore                   rni

misc #