Search the Asterisk Blog

Using DEBUG_THREADS to find deadlocks.

By Richard Mudgett

Asterisk’s DEBUG_THREADS is a compile time tool that helps find deadlocks involving Asterisk locks. You enable DEBUG_THREADS in menuselect’s “Compiler Flags” menu along with other useful compile time options like DONT_OPTIMIZE and BETTER_BACKTRACES. It is strongly recommended that you enable BETTER_BACKTRACES for the output of the Command Line Interface (CLI) “core show locks” command to be really helpful.

Using the ast_coredumper script which is described in Getting-a-Backtrace is an easy way to gather the locking information also presented by the CLI “core show locks” command and to gather backtraces.

Examining a deadlock issue

We’ll examine the ASTERISK-27460 CDR deadlock issue filed against Asterisk 13.18.3. Below is the output of the CLI “core show locks” command.

In a classic deadlock situation we have two threads holding different locks and these threads are waiting to acquire the lock that the other thread has. With a classic deadlock the CLI “core show locks” command output is enough to tell us which locks are involved and how the code got there. As we can see in the CLI “core show locks” command output we don’t have a classic deadlock. Thread LWP:27257 is waiting for the 0xa75f588 lock that is held by thread LWP:27463 but thread LWP:27463 is not waiting on any lock. We need more information.

Looking at the threads identified by the CLI “core show locks” output in the backtrace files we see that thread LWP:27463 is waiting on a condition variable. This particular condition variable does not get set until thread LWP:27257 completes its current task. This then is the deadlock scenario. Thread LWP:27463 is waiting on thread LWP:27257 to complete its task but thread LWP:27257 is waiting on a lock held by thread LWP:27463.

Once the deadlock scenario is identified a means to prevent the deadlock needs to be devised. In this case the CDR dialplan function was used in a deprecated manner to gather information readily available using the CHANNEL dialplan function. The Asterisk issue has links to the merged gerrit review patch that fixed the issue.

Summary

DEBUG_THREADS is by no means a silver bullet. It is not recommended to normally run Asterisk with the option enabled as it can really impact performance. It does not help as much in non-classic deadlock scenarios or if the locks involved are in a third-party library such as PJPROJECT or an ODBC driver. In those cases you need backtraces more than the output of the CLI “core show locks” command. However, it does readily show the Asterisk locks a thread has and wants which is difficult to determine from just a backtrace.

For more information about locking in Asterisk see Locking-in-Asterisk.

No Comments Yet

Get the conversation started!

Add to the Discussion

Your email address will not be published. Required fields are marked *

About the Author

Richard Mudgett

Richard Mudgett is a Senior Software Developer at Digium. While a prolific developer and contributor to Asterisk, he's elusive and can be difficult to spot outside of his native #asterisk-dev environs. We were impressed we got him to write a blog post.

See All of Richard's Articles

More From
The Digium Blog

  • No items