Am 17.06.2016 um 10:36 schrieb Damien Buhl:
Hi Klemens,
We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function.
The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux.
Without the use of vfork, you end up in situations where you get the following error: `boost::process::detail::posix_start: fork(2) failed: Cannot allocate memory`
I think changing from fork to vfork is not much and brings alot of advantages, but one must be aware that at_fork handler won't be called. But this is not important as fork is used to do execve afterwards in Boost.Process. Hi Damien,
I appreciate your problem, but I am not sure it's that easy. There's on major problem: vfork is removed from the posix standard as of POSIX.1-2008. And I'm trying to conform to posix, not linux.
So I guess, I wouldn´t use vfork as default, but it might be possible to add a property, which will cause the library to use that. I.e. you write:
boost::process::child c("java.exe", "overhead.jar", boost::process::posix::use_vfork);
But I'd need to be able to check if vfork is available, so I can disable the property if not.
Would that be sufficient for your problem? Naturally if the code still compiles on windows even though I'm using boost::process::posix::use_vfork(_if_possible) then yes for me all will be fine. But on the other hand from a library design point of view, shouldn't the
On 16/06/2016 18:02, Klemens Morgenstern wrote: library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me. Again: obsolete and now removed in the posix-standard and that's what I ought to go with. Please note, that I'm trying to provide two platforms: Posix & Windows. Not Linux & Windows. Thereby I want to provide the most common way for both platforms; and though I really appreciate your scenario, I would not consider it the common way.
On 17/06/2016 01:21, Gavin Lambert wrote:
Is this specifically for NOMMU linux?
MMU architectures shouldn't have this issue, as fork does a shared-memory-copy-on-write mapping so that the pages aren't actually duplicated unless written to, and the subsequent exec* unmaps the pages so this never happens (other than a bit of stack).
NOMMU architectures don't support those kinds of mappings, so have to be rewritten to use vfork instead (which is generally unsafe unless immediately followed by exec*, unless you know what you're doing).
Although the last time that I played with NOMMU, fork used to just fail; perhaps it's been changed to make copies of all the pages instead? If so, that would indeed be problematic for large parent processes. Hi Gavin ;)
It's with an MMU but the parent process on this product is a memory-hungry-monster-jvm and the fork fails. It seems a bit strange to me, that you would use boost.process here, instead of - you know - java.io.process. But I guess you want
I can be wrong, and it looks like you know more from what's happening there, but as long as I can remember the issue arose due to virtual memory commit, because fork on linux even though is optimized to not copy the whole process pages but to do copy-on-write, still needs to commit virtual memory, and overcommitment is not always allowed, or by default is heuristically allowed. That's probably right, because elsewise you'd get an error in the forked
I think you underestimate the dangers of vfork; because you know, two processes sharing memory (including the stack!) can be rather fun. There's a reason it was removed and so it will be optional in boost.process. It would be a platform extension, so no, this would NOT compile with windows. I thought about turning the platform extensions into NOPs , but that's just too weird - especially since things like signal(SIGCHLD, ...) are also provided. There you should rather use the preprocessor and put an #ifdef there - then it's obvious what you're doing. Also I'd need a #define to know whether vfork is available, that I would probably provide. So you could then write something like that: child c("ls" #if defined(BOOST_POSIX_HAS_VFORK) , posix::use_vfork #endif ); I guess I can check that via 'CLONE_VFORK'. performance and your java-developers want a job, so JNI is the way to go? process and there would be no way of getting to know it.