Asterisk’s MALLOC_DEBUG is a compile time tool that helps find some common memory usage problems such as memory corruption and memory leaks. The debug tool helps find similar things as valgrind but without the huge performance penalty. You enable MALLOC_DEBUG in menuselect’s “Compiler Flags” menu along with other useful compile time options like DONT_OPTIMIZE.
With MALLOC_DEBUG enabled, recompile, install Asterisk, and reproduce your problem. If MALLOC_DEBUG finds anything it logs the findings to the mmlog file in the default /var/log/asterisk location. If it doesn’t find anything the mmlog file will just contain “New session” lines indicating when Asterisk was started.
MALLOC_DEBUG does many things for you.
- It keeps track of each block of memory Asterisk allocates until it is actually freed when it is pushed out of a fixed size freed memory holding queue. It remembers where a block of memory was allocated and how large it is.
- It checks freed memory for writes above and below the block of memory. If it sees the guard pattern has changed it complains of high and low fence violations respectively.
- It fills freed memory blocks with the 0xdeaddead pattern. The memory is then placed into a holding queue for awhile.
- If you get a crash because of a pointer trying to access the 0xdeaddead (or 0xdeaddeaddeaddead) fill pattern address then you know someone was using a freed block of memory. The pattern was specifically chosen to cause a crash on most machines and be obvious for this reason.
- It complains of memory corruption when a memory block is pushed out of the holding queue and the fill pattern has changed.
- It can give you live allocation reports using the CLI “memory show summary” command. You can then watch memory leaks consuming more and more memory.
- It can give you an unreleased allocation list on Asterisk shutdown when requested.
Finding memory corruption
High and low fence violations are direct memory corruption bugs. In this case the corruption happens while the block is still allocated. For fence violations you know that someone wrote just outside the block of memory, which end of the block was overwritten, and where it was allocated. With this information you have to look at the code for places where it could write outside the block. High fence violations are usually caused by a miscalculation of the memory buffer size needed. Another similar cause of a high fence violation is a missing or incorrect check for enough room remaining before inserting something into the buffer.
Bugs that cause the memory corrupted after free warning are difficult to find. In this case you know which block was corrupted after it was freed and where it was allocated. The problem is most likely caused by using a stale pointer to write to the memory block after it was freed. You have to look at the code handling the memory block for writes after a free.
Bugs that cause the crash on 0xdeaddead have the best information to get them fixed. Be happy if you get this crash when you are searching for a memory corruption bug. For this crash, the backtrace shows you exactly where freed memory is used so it is a matter of figuring out why the code is using a stale pointer to freed memory.
Finding memory leaks
Memory leaks are shown by a lot of allocations from an allocation location that are never released. The larger the leak the more allocations that are never released. You can narrow it down by noting which allocations don’t seem to be freed because the number of the allocations from a location keep increasing.
Simply monitor memory usage using the CLI “memory show summary” command and note which files show an increase in memory consumption or allocations. Once you have identified files leaking memory, you can see which lines in that file are allocating the memory. So much for the easy part. Now you need to use other symptoms of the memory leak to determine what to do with the information MALLOC_DEBUG provides. You need to look at the Asterisk debug logs to see what the system is doing. You may even need to add more debugging log messages to trace the activity of the code handling the leaked memory.
If you are using the “top” application to monitor the Asterisk process memory allocation, you need to be aware that memory is not returned to the system if the libc heap allocator obtained the memory even if Asterisk itself returned the memory to the heap.
The CLI “memory atexit summary” commands allow you to see what memory is still allocated when asterisk shuts down. If you are looking for memory leaks on Asterisk shutdown then you need to use the CLI “core stop gracefully” command. Asterisk cleans up as much as it can before exiting on a graceful shutdown. Using the CLI “core stop now” command causes Asterisk to do the absolute minimum cleanup before exiting so you will see a lot of unreleased memory.
MALLOC_DEBUG is by no means a silver bullet. However, it does give very useful information when tracking down memory usage bugs.
No Comments Yet
The Digium Blog
- No items