[unordered] valgrind reports "possibly lost" memory when not destroying the container
Hi! I have a problem with containers from Boost.Unordered. Before I go into details please let me show you the big picture. I have a code that creates "unordered" containers in a lot of places, but these containers never really get destroyed / deleted at the end of the program. However, they are all "still reachable" through various pointers in the program and... valgrind reports them as such. This is OK for me. Now the problem is that the containers from Boost.Unordered do something clever with pointers and valgrind also reports some "possibly lost" memory blocks [1]. This only happens when the unordered containers are not destroyed before the end of the program. When I do the proper clean-up valgrind stays quiet. However, in the code base where I discovered the problem it is currently not possible to do such a clean-up. The problem became serious for me for a number of reasons. In the code base that I referred to above valgrind reports ~7000 "possibly lost" memory blocks. Some of them come from Boost.Unordered containers. I could try to suppress them via a suppressions file, but the problem is that the Boost.Unordered code gets inlined in nearly all cases and there's no simple way to suppress valgrind from reporting these "possibly lost" memory blocks. Without that I'm stuck with tons of reports that I can't really do anything about. Given the above I have a few questions: - has anyone ever encountered this problem? has anyone solved it? - is it possible to force some simpler but maybe more expensive memory management in Boost.Unordered to suppress these "possibly lost" reports? - what other options do I have? My setup is: - gcc 4.8.2 - Boost 1.55 - Ubuntu 14.04 x86_64 WBR, Adam Romanek [1] http://pastebin.com/A4fvx0K0
On 7/4/2014 02:18 AM, Adam Romanek wrote:
- is it possible to force some simpler but maybe more expensive memory management in Boost.Unordered to suppress these "possibly lost" reports? - what other options do I have?
If you have a problem where the containers are consuming too many resources, than you should consider ways to release them. Valgrind is just doing its job in reporting these blocks. They are in use when the program exits. Memory management isn't going to change that. Perhaps you could write a parser that could digest the valgrind outputs and filter out these warnings? Bear in mind though that these warnings could be useful if you have a long running process. Sometimes these things can build up over time. Frankly, it seems like it would be easier to manage these such that they are destroyed properly on program exit (read as not a crash.) HTH Brandon
On 04.07.2014 16:01, Brandon Kohn wrote:
On 7/4/2014 02:18 AM, Adam Romanek wrote:
- is it possible to force some simpler but maybe more expensive memory management in Boost.Unordered to suppress these "possibly lost" reports? - what other options do I have?
If you have a problem where the containers are consuming too many resources, than you should consider ways to release them. Valgrind is just doing its job in reporting these blocks. They are in use when the program exits. Memory management isn't going to change that.
I'm not complaining that the containers use too much memory. The problem is different. I'm perfectly aware of the fact that releasing the containers would make valgrind produce clean reports. But as indicated in the original post, I can't reliably make the whole application shutdown with proper clean-up. It hasn't been designed for this from the start and now it would be a huge work to implement all this properly. We've already tried this and we ended up with lots of deadlocks and other issues. Valgrind categorizes memory leaks into 4 groups: definitely, indirectly and possibly lost + still reachable. We can easily eliminate leaks from the first two groups. The "possibly lost" is not that easy, as the memory is still reachable through some pointers but not directly. This is the case with unordered containers. Instead I'd like valgrind to see them as "still reachable" which is the category that we don't really pay attention to. We know that we don't release the memory at the end of the program so this is the memory in use, the memory that we need (or we think we need). But still, "possibly lost" memory is something we can't ignore as it may indicate a regular memory leak.
Perhaps you could write a parser that could digest the valgrind outputs and filter out these warnings? Bear in mind though that these warnings could be useful if you have a long running process. Sometimes these things can build up over time.
As indicated in my previous post, this is not possible. The calls through Boost.Unordered get inlined and they are simply not observable in non-debug builds.
Frankly, it seems like it would be easier to manage these such that they are destroyed properly on program exit (read as not a crash.)
It's not easy to make a highly multithreaded program release all its resources when it ends. Especially when the program hasn't been designed for such use from the start. I understand that the issue I'm reporting is not a bug. However solving it would help in maintenance a lot. WBR, Adam Romanek
participants (2)
-
Adam Romanek -
Brandon Kohn