Yeah, was my first idea too, but I checked that. My posted code is only a summary of the essential code passage, but i also tried to allocate and deallocate the memory multiple times, hoping, that the C++ runtime still "owned" the memory and hence would be able to manage the second, third, ... and so on allocation, without acquiring more memory from the OS, but every allocation took the same amount of memory from the OS. -- View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind... Sent from the Boost - Users mailing list archive at Nabble.com.

On Thu, 28 Apr 2016 12:14:00 -0700 (PDT) Norman <kradepon@googlemail.com> wrote:

Alright, i made an example, that shows my problem:

http://melpon.org/wandbox/permlink/S56tYueJySIHdsWO

This does not show any problem, although I am uncertain on the intent of the `async_accept` calls. The only non-completed handler in the "during" stage should be the async_wait, however there is no guaranteed sequencing between the acceptor handlers and the async_wait handler. The counter indicates all of the other callbacks have completed in my executions of the code. The during and after printing only indicate the RSS is the same at those two execution points. This does not necessarily indicate a problem. In fact, the following `stop` call could result in a RSS increase (additional code page mapping), and then the RSS could decrease to the prior level after the `async_wait` and `run` complete and free any used memory. I simply do not see any information learned from this example.

Vinnie_win aka. Vinnie falco told me, that I might be looking at the wrong indicator for my memory use. But I thought the Resident Set Size is the value to look for, since I want to know how much memory my program is currently allocating.

Any help would still be really appriciated :-)

The RSS value indicates the number of virtual pages that have a physical page mapping for the process. There is no guaranteed correlation between the amount of memory requested via new/delete OR the amount of memory being managed by the C++ runtime. "Allocated" memory being managed by the C++ runtime can be swapped to disk which lowers the RSS but not the amount of memory being managed by the runtime. New code segments can be executed which requires additional pages of the mmap'ed executable or shared library to be copied to physical memory which will increase the RSS but not the amount of memory being managed by the C++ runtime. A large allocation into the C++ runtime can be fulfilled with a single anonymous mmap which delays any RSS increase until the page(s) have been "touched". As an example (stealing your getUsedMemory function): #include <fstream> #include <iostream> #include <memory> #include <string> #include <boost/algorithm/string.hpp> std::string getUsedMemory() { try { std::ifstream inFile("/proc/self/status"); if (!inFile.good()) { std::cout << "Cannot open status file." << std::endl; } std::string line; while (std::getline(inFile, line)) { if (boost::starts_with(line, "VmRSS:")) { inFile.close(); return std::string(&line[7], line.size() - 7); } } inFile.close(); } catch (std::exception& e) { std::cout << " Exception: " << e.what() << std::endl; } return "Unknown"; } int main() { std::unique_ptr<char[]> bytes(new char[50*1024*1024]); // 50 MiB std::cout << "Used memory: " << getUsedMemory() << std::endl; return 0; } That is unlikely to print 50+ MiB of "used memory". Wandbox (Gcc 5.3) prints: Used memory: 5968 kB And my local box prints even less. It is probably easier to use the tools I mentioned earlier or something like massif if you are trying to track general process memory usage or leaks. Lee

Hi, The exact behaviour depends on your run-time implementation. It is most likely not very much memory that the run-time is claiming. Have you seen any issue in practice? Or is it a theoretical problem like "when do I get my 2 MB of my 64GB memory back?" On 2016-05-10 14:40, Norman wrote:

I understand, that the RSS is not a direct indicator for 'unused' memory. Since I obviously have problems to clearify my problem, i made another example:

Converging memory consumption example <http://melpon.org/wandbox/permlink/i5k59o38K8azDX2o>

In this example, the amount of memory beeing allocated by the programm converges to a limit, which shows exactly what you told me. So the C++ runtime holds on to some allocated memory, even though the objects have been 'freed', meaning that the application does not need to allocate more memory for every function call.

My problem with this behaviour is, that my application should run for an infinit amount of time. It's a server application, which allocates a greater amount of memory in between. So if this memory is allocated for a little while, even when the memory is 'not in use' anymore, that's fine with me. But at some time, the memory needs to be released to the OS. And that's exactly where my problem appears. I don't know, if the memory will ever be reclaimed by the OS. Of course, RSS does not tell me how much memory my application is currently using, but it tells me at least the minimum amount of memory currently managed by the C++ runtime. And if this increases to a huge amount, my OS does not have access to this memory, right?

So my remaining question is, if the ioService object will (or more precisly the C++ runtime) will ever release the allocated memory to the OS or if i have to do this manually.

-- View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind... Sent from the Boost - Users mailing list archive at Nabble.com. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

My first approach was to search for leaks with Valgrind and i didn't find any. Also i never mentioned any leaks, so that's not my concern. The only problem i had/have is, that the RSS is not decreasing during runtime, even after a long time when my application is not using the allocated memory anymore. For instance i connected 30.000 dummy clients to my server, so the memory consumption went up to ~ 50% of my available memory. Then i disconnected them (leading to a lot of destructed objects), which should lead to a lot of freed memory. But my application still claims 50% of my RAM, according to RSS. If i now connect another 30.000 clients, my RSS changes only marginally. But i cannot understand, why my application never releases any of the claimed memory to the OS. It feels like there is some kind of a capacity inside the application (a map-like container for instance), which can only increase and will never decrease. Since its capacity does never decrease, the claimed memory won't be released. @Oswin Krause It's more of a practical concern. Since i never observed my OS 'reclaiming' the virtual pages, i don't know if this will ever happen. All i observed, is that my application claimed some memory and never released it (or at least only an alternating small portion of it). -- View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind... Sent from the Boost - Users mailing list archive at Nabble.com.

Alright, let's asume you are right. Can you explain the following behaviour: i create a vector<string> and fill it with a huge amount of strings, so it takes up to ~50% of my RSS. After the vector is filled, i observe the RSS and as expected, it's quite big. Then i clear the vector and let it run out of scope and something special happenes: The RSS is decreased by a significant amount (close to what it was before i created the vector)! This should not happen in your scenario, right, since the OS does not need the memory, but it still does get reclaimed by the OS. So i'm really sry, but this procedure, where memory is sometimes hold on to and sometimes not does make no sense to me. I do understand that the OS memory management tries to optimize the access onto the memory and hence not all memory might be reclaimed right away, but not in this scale. I've never seen programs holding on to memory forever until the OS reclaims it. And im not talking about a few bytes here and there, i'm talking about hundreds of megabytes. By the way, the behaviour, described with a vector<string> also works with a vector<shared_ptr<sslSocket>>, as long as no assynchronous job, depending on the sslSocket, has been given to the acceptor/ioService. -- View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind... Sent from the Boost - Users mailing list archive at Nabble.com.

It took me a while, but i finally managed to work it out by myself. *So to clearify things, let's make sure, that the root of my problem is understood:* I'm developing a server application, which is meant to run for an infinit amount of time. This application must be able to handle a lot of concurrent incomming connections. At some point in time, there may be a peak in load, leading to a lot of claimed memory to my application. Then after a while, most incomming requests have been processed, causing a lot of objects to be freed in runtime. Since the OS is in no need for memory (My application is the only huge memory consumer on the server), all freed memory stays with my application and can be reused at another point in time. This would be absolutely fine with me, but some customers and administrators might misinterpret the greater amount of constantly claimed memory as a memory leaking application. To avoid this, i wanted to hand some of the unused memory back to the OS manually. In my example, the bound handlers to the *ioService* (e.g. accepting a new connection) would be freed in runtime, but the appropriate memory won't be reclaimed by the OS. So to do this manually, i found the following solution: *Release unused heap memory under Linux in C/C++: int malloc_trim(size_t pad)* The documentation can be found here <http://man7.org/linux/man-pages/man3/malloc_trim.3.html> . Simplified explained, this method releases unused memory from the heap to the OS, which is exactly what i've been searching for. I'm aware that under memory optimization aspects, the manual use of this function maybe dangerous, but since i only want to release the memory every few minutes, this performance issue is acceptable to me. Thank you all for your efforts and patience! -- View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind... Sent from the Boost - Users mailing list archive at Nabble.com.

On 28 Apr 2016 at 12:14, Norman wrote:

Alright, i made an example, that shows my problem:

http://melpon.org/wandbox/permlink/S56tYueJySIHdsWO

Vinnie_win aka. Vinnie falco told me, that I might be looking at the wrong indicator for my memory use. But I thought the Resident Set Size is the value to look for, since I want to know how much memory my program is currently allocating.

Any help would still be really appriciated :-)

You appear to not understand how virtual memory and the system memory allocator works and interact with one another. Please read the papers from Denning et al from the 1970s. All major OSs use his design. RSS is the *cache* of memory held by the system in RAM of your process' memory. It is entirely dependent on system load, how much caching your kernel chooses, and what the libraries your processes do internally. It has little relevance to "how much memory my program is currently allocating". The closest proxy to such a thing - which doesn't really exist on any major OS of the past thirty years - is probably the dirtied page count. And even there, the system memory allocator will intentionally cache a large quantity of memory it fetches from the OS, so just because you free() everything you malloc() doesn't mean any memory is released to the system. If you really, really, really want to always free everything you malloc, use mmap() to allocate and munmap() to free, or even use sbrk(). Note that your program will run four orders of magntitude slower. tl;dr; If this really bothers you, use a custom STL allocator. Otherwise so long as dirtied pages doesn't keep rising over time, it's not an issue and don't worry about it. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/