rr (short for “record and replay”) is a very powerful debugging tool for C++ programs, or programs written in other compiled languages like Rust1. It’s essentially a reverse debugger, which allows you to record the execution of a program, and then replay it in the debugger, moving forwards or backwards in the replay.
I’ve been using rr for Firefox development at Mozilla, and have found it to be enormously useful.
One task that comes up very often while debugging is figuring out why a function produced a particular value. In rr, this is often done by going back to the beginning of the function, and then stepping through it line by line.
This can be tedious, particularly for long functions. To help automate this task, I wrote – in collaboration with my friend Derek Berger, who is learning Rust – a small rr plugin called Control Flow Visualizer, or CFViz for short.
To illustrate CFViz, consider this example function foo()
and a call site for it:
With the CFViz plugin loaded into rr, if you invoke the command cfviz
while broken anywhere in the call to foo()
during a replay, you get the following output:
Basically, the plugin illustrates what path control flow took through the function, by coloring each line of code based on whether and how often it was executed. This way, you can tell at a glance things like:
- which of several
return
statements produced the function’s return value - which conditional branches were taken during the execution
- which loops inside the function are hot (were executed many times)
saving you the trouble of having to step through the function to determine this information.
CFViz’s implementation strategy is simple: it uses gdb’s Python API to step through the function of interest and see which lines were executed in what order. In then passes that information to a small Rust program which handles the formatting and colorization of the output.
While designed with rr in mind, CFViz also works with vanilla gdb, with the limitation that it will only visualize the rest of the function’s execution from the point where it was invoked (since, without rr, it cannot go backwards to the function’s starting point).
I’ve found CFViz to be quite useful for debugging Firefox’s C++ code. Hope you find it useful too!
CFViz is open source. Bug reports, patches, and other contributions are welcome!
Footnotes
1. rr also has a few important limitations: it only runs on Intel CPUs, and only on Linux (although there is a similar tool called Time-Travel Debugging for Windows)
This is really cool!
Thanks! It was inspired by some of the server-side visualizations that you were mentioning at the San Francisco All Hands. I figured I could do a little bit of that locally via a plugin like this.
Ah, I thought it might have been 🙂
I discuss using the same technique to debug the processing of Firefox Telemetry data at 15:00-18:00 of this talk: https://youtu.be/QHJBPzfrokU?list=PLYx7XA2nY5GfdAFycPLBdUDOUtdQIVoMf&t=993