Hey. Let's do some debugging today.
Students often have trouble debugging their programs, but things can get
especially difficult when we have multiple threads involved. And, lately
I've been talking a lot about threads— creating threads, manipulating threads
multi-threaded programming, and some of you have asked for some help with
debugging these programs. And so, today I thought we would look at how we could go
about debugging a multi-threaded program. Now for today's video I'm assuming a few
things. I'm assuming that you have a basic understanding of threads and of
the pthreads library, and I'm also assuming that you have some basic
understanding of how to use gdb to debug programs. If that's not the case, I've got
a bunch of videos that can help. Just pause this video. I'll link to a bunch of
those videos in the description. Go watch those, and then come back and this will
make a whole lot more sense. So, for today's video we're gonna look at the
last example from my last video. This example has two threads: one counts to a
billion, and one uses a socket to go out and download a web page. Pretty simple,
but it's a useful example. Okay, so let's load our program into gdb and try to
trace its execution. Let's break at the start of main. We'll get it started—
okay right here we find ourselves at the point at which our example
creates the second thread. Now, when I use the "next" command to step along in the
main thread, you notice that the counter increases because each time I run the
"next" command the new thread that I created—the other thread is run for
a while and then it stops when the main thread is stopped. The thing is, when
I keep running it you see different behaviors, like different
amounts of execution. And, this is one reason why you might get confused when
you're debugging multi-threaded programs is that you have one thread that you
seem to have control over and the other thread that just kind of does whatever
it wants. So, what can we do? Okay, so the first thing we can do is use the
"info threads" command. That shows us what threads are running and which thread
we're currently stepping through. Okay, so, this is helpful just to get a bearing
for where we are. So, I can see in this case—I can see the main thread and the
new thread that I created. I can see what line they're each on, and the asterisk tells
me which one I'm currently debugging. Currently, like the current active one.
Okay, so, also note that gdb gives each thread a thread ID, and just in case
you're wondering what LWP stands for, that stands for
"LightWeight Process", which is just another name for threads. And, I can
change which thread I'm stepping through by using the "thread" command and just
give it the ID for the thread I want. So now I'm on thread 1. Now I'm on thread 2.
And now, I can step through the code in thread 2. So, this gives me a little more
control over the threads that I'm stepping through, but you notice that I'm
still jumping into the threads at arbitrary points in time. So, typically,
when I'm debugging a multi-threaded program, I also stick breakpoints at key
points either at the beginning of the thread function or at the point in the
thread function that I actually want to debug—that I actually care about—and
that way I know that it's not gonna get beyond that point. And, I'm gonna have an
opportunity to switch over into that new thread. So, in this example I might set a
breakpoint at the start of the "count_to_big" function—that's my thread function.
So, I just set a breakpoint right there, and so then I get stopped right at the
beginning. And now, I can see basically from the beginning what happens in that
thread. And, of course, this function does count a lot so let's say that there's
something interesting that happens when the counter reaches 4,500 (actually 0x4500) or some
arbitrary number. So, if that's the case then I can use a watchpoint to stop
whenever the counter reaches 4,500. And, this will stop execution, regardless of
which thread actually incremented that counter and which thread caused it to
reach 4,500. It's just basically going to watch that value, and anytime that it
reaches 4,500, it's gonna stop us just like this. And, that's really useful if I
have a program that has a bunch of different threads that are each
manipulating memory and I'm not really sure how I'm getting to the point that
I'm getting to. Some value gets written to a certain variable, and I don't know
how it got there, which thread wrote it there, or what's going on. I can just use
a watchpoint, and it's going to stop execution whenever that happens and I
can look around and see what state each of my threads are in. Okay, so, I can also
set breakpoints that are specific to a single thread. In this case, you just use
the "break" command like you would normally but you add the word "thread" and
the thread ID you want at the end. GDB also lets me apply a command to
multiple threads. The most common reason that I use this is if I want to
see a stack trace for all my threads. I can just type "thread apply all bt" or
"backtrace". I could write out the whole backtrace command if I want, and
that's going to be like me calling backtrace on every single one of my threads.
And, this is super useful if I'm not really sure where I'm at. I've just
stopped at a watchpoint or a breakpoint, and I just want
to get a snapshot of what's going on in my program at that time. I can just get a
backtrace on all the threads, see what function they're in, and how they got
there. And, that's super useful. Now this thread apply all works for any
command. It's not just backtrace. I could use it for printing out the stack
frames or navigating the call stack or anything like that.
Basically any command—this will just apply that command to all the different
threads. I'm sure I'm missing a command or two, but this is how I debug
multi-threaded programs in GDB. Just a few commands. They allow me to switch
between threads. They allow me to inspect each thread and set watchpoints and
breakpoints, and so I can stop at key points during my program's execution. And,
that's it for today. Short and simple. But, I hope it's useful. I hope this helps
you on your next multi-threaded program to find your bugs more quickly and know
that you've got a few tools that not everybody knows about in GDB that can
help make your debugging process less painful and faster. And, so, until next
time happy coding, and I'll see you soon.