[boost.process] 0.6 Alpha

older
Checking interest in stacktrace...

Klemens Morgenstern

4 Jun 2016 4 Jun '16

7:52 p.m.

Hi all, the boost.process 0.6 library is now in alpha, that is: it has some (compiling) examples, a documentation and tests that pass on windows as well as on linux. It basically allows you to manage a process like this: pstream pipe; child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in < "data.csv", std_out > pipe, std_err > null, start_dir="./bin"); std::string data; pipe >> data; c.wait(); cout << c.exit_code() << endl; You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process If you want to use it on windows, you currently would need to clone my boost.winapi fork also: https://github.com/klemens-morgenstern/winapi As for the scope of the library, it is meant to implement a small, portable layer to allow platform-independent process management. Thereby it's functionality is (unless I missed something) basically what is possible on posix as well as on windows. There are some platform extension, but I would not recommend to use them. It is also possible (though not yet documented) to write extensions, and if they work on both platforms I will gladly add them. It is however not the goal of the library, to be the be-all and end-all of process management; in past disscussions, different ideas where brought up, like implementing a DSEL for that or only allowing asynchronous communication. This library will never be that, since it would be much more fitting to build this on-top of it. E.g. a DSEL could be written (e.g. boost.shell) using boost.process underneath, so the library could concentrate only on the DSEL. It doesn't depend on boost.iostreams anymore, though it still uses boost.asio, boost.fusion and boost.filesystem. As for the open issues: some error handling is still lacking (but that'd be something like being unable to duplicate a handle), which would be required for proper extension, which is also why it's not documented. Also there's still an open discussion about the naming of the class and functions (child, system, spawn), so that may change also. But I consider this simple enough for a easy refactoring. If anyone has naming ideas, please tell me. One Idea was 'subprocess' but 'process::subprocess' just looks too strange to me. I hope you try it out, and give me some feedbacks. Or write angry error reports, both helps. Thanks! Klemens

Show replies by date

Edward Diener

4 Jun 4 Jun

9:41 p.m.

On 6/4/2016 3:52 PM, Klemens Morgenstern wrote:

...

Hi all,

the boost.process 0.6 library is now in alpha, that is: it has some (compiling) examples, a documentation and tests that pass on windows as well as on linux.

It basically allows you to manage a process like this:

pstream pipe;

child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in < "data.csv", std_out > pipe, std_err > null, start_dir="./bin");

std::string data; pipe >> data;

c.wait(); cout << c.exit_code() << endl;

You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process

I would like to immediately suggest that when you show a code example with no explanation of what it is doing that you put comments in the code example. This should be true whether the code example is in a message or in your docs.

Klemens Morgenstern

10:01 p.m.

...

...
You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process

I would like to immediately suggest that when you show a code example with no explanation of what it is doing that you put comments in the code example. This should be true whether the code example is in a message or in your docs.

You're of course right. I've worked too long on this, so it all seems obvious to me. I'll add some more detailed example in the documentation as soon as I have the time. As for the previous example (since I can't edit the mail): pstream pipe; //iostream implementation with a pipe //start the process child c( "some_prog", //program name "--some-arg", //argument env["PATH"]+="C:/stuff", //add C:/stuff to the PATH-environment variable std_in < "data.csv", //redirect stdin to a file std_out > pipe, //redirect stdout to created pipe std_err > null, //redirect stderr to /dev/null start_dir="./bin");//start in ./bin std::string data; pipe >> data; //read a string from the pipe. c.wait(); //wait for the program to exit cout << c.exit_code() << endl;//print the return code

Rob Stewart

6 Jun 6 Jun

12:07 a.m.

On June 4, 2016 6:01:52 PM EDT, Klemens Morgenstern wrote:

...

...
...
You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process

I would like to immediately suggest that when you show a code example with no explanation of what it is doing that you put comments in the code example. This should be true whether the code example is in a message or in your docs.

You're of course right. I've worked too long on this, so it all seems obvious to me. I'll add some more detailed example in the documentation as soon as I have the time. As for the previous example (since I can't edit the mail):

pstream pipe; //iostream implementation with a pipe

//start the process child c( "some_prog", //program name "--some-arg", //argument env["PATH"]+="C:/stuff", //add C:/stuff to the PATH-environment variable std_in < "data.csv", //redirect stdin to a file std_out > pipe, //redirect stdout to created pipe std_err > null, //redirect stderr to /dev/null start_dir="./bin");//start in ./bin

std::string data; pipe >> data; //read a string from the pipe.

c.wait(); //wait for the program to exit cout << c.exit_code() << endl;//print the return code

I inferred the precise meaning from the code before you added the comments, which implies that it's intuitive (to me at least). ___ Rob (Sent from my portable computation engine)

Klaim - Joël Lamotte

9:13 a.m.

On 4 June 2016 at 21:52, Klemens Morgenstern wrote:

...

Hi all,

the boost.process 0.6 library is now in alpha, that is: it has some (compiling) examples, a documentation and tests that pass on windows as well as on linux.

It basically allows you to manage a process like this:

pstream pipe;

child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in < "data.csv", std_out > pipe, std_err > null, start_dir="./bin");

std::string data; pipe >> data;

c.wait(); cout << c.exit_code() << endl;

You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process

If you want to use it on windows, you currently would need to clone my boost.winapi fork also: https://github.com/klemens-morgenstern/winapi

As for the scope of the library, it is meant to implement a small, portable layer to allow platform-independent process management. Thereby it's functionality is (unless I missed something) basically what is possible on posix as well as on windows. There are some platform extension, but I would not recommend to use them. It is also possible (though not yet documented) to write extensions, and if they work on both platforms I will gladly add them.

It is however not the goal of the library, to be the be-all and end-all of process management; in past disscussions, different ideas where brought up, like implementing a DSEL for that or only allowing asynchronous communication. This library will never be that, since it would be much more fitting to build this on-top of it. E.g. a DSEL could be written (e.g. boost.shell) using boost.process underneath, so the library could concentrate only on the DSEL.

It doesn't depend on boost.iostreams anymore, though it still uses boost.asio, boost.fusion and boost.filesystem.

As for the open issues: some error handling is still lacking (but that'd be something like being unable to duplicate a handle), which would be required for proper extension, which is also why it's not documented. Also there's still an open discussion about the naming of the class and functions (child, system, spawn), so that may change also. But I consider this simple enough for a easy refactoring. If anyone has naming ideas, please tell me. One Idea was 'subprocess' but 'process::subprocess' just looks too strange to me.

I hope you try it out, and give me some feedbacks. Or write angry error reports, both helps.

Thanks!

Klemens

...

So far I had time only to go quickly through the documentation and a bit in the code so here are quick remarks: 1. It is not clear at all what happen when child object is destroyed. 2. The way child process termination is implemented for each platform should be documented. 3. On most platforms there is a way to send a "quit" command to the process and a way to kill it by force ("kill -9"). Clarifying what is happening in the default cases and having a way to terminate one way or the other (or a combination of both, that is requesting termination but force-kill after a time if it didn't die) On windows there is an issue with this because the "termination request" message can only be received if you didn't build a console application but a windows application (with a WinMain) but that only means that the message could be ignored by the process if it's console application (there are ways to have both WinMain and a console window, it's just a bit more clumsy to setup). In our (Softbank Robotics EU) use of child process tracking we need both termination request and forced. One could argue that the termination request do not need to be implemented by this library but it would be very useful to have a cross platform function doing this for simple cases. Joël Lamotte

Klemens Morgenstern

10 a.m.

...

So far I had time only to go quickly through the documentation and a bit in the code so here are quick remarks:

1. It is not clear at all what happen when child object is destroyed. Oh, ok that's only written in the reference atm. It terminates the child

process if it wasn't detached or exited before. I added that, will be up in the next doc build.

...

2. The way child process termination is implemented for each platform should be documented. Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added)

...

3. On most platforms there is a way to send a "quit" command to the process and a way to kill it by force ("kill -9"). Clarifying what is happening in the default cases and having a way to terminate one way or the other (or a combination of both, that is requesting termination but force-kill after a time if it didn't die) On windows there is an issue with this because the "termination request" message can only be received if you didn't build a console application but a windows application (with a WinMain) but that only means that the message could be ignored by the process if it's console application (there are ways to have both WinMain and a console window, it's just a bit more clumsy to setup).

In our (Softbank Robotics EU) use of child process tracking we need both termination request and forced.

One could argue that the termination request do not need to be implemented by this library but it would be very useful to have a cross platform function doing this for simple cases. From how I understand it, you send the terminate request do the HWND handle, not the proc handle. That technically means, that you cannot tell the process to exit, but a part of it. Hence I don't consider this the same as signaling the process to exit, which means it's not portable, thus I do argue it shouldn't be part of the library. If I have a child::request_exit function, it must work on all processes, not just some of the windows-procs. If there was a portable way to do it, it would be part of this library. I think it's easy enough, you can get the pid (or even gid, if you use a group) and implement that in two lines. On windows you can get the HWND as an std::intptr_t through a pipe, though that's a bit more code.

Thanks, Klemens

Klaim - Joël Lamotte

10:49 a.m.

On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...

So far I had time only to go quickly through the documentation and a bit in

...
the code so here are quick remarks:

1. It is not clear at all what happen when child object is destroyed.

Oh, ok that's only written in the reference atm. It terminates the child process if it wasn't detached or exited before. I added that, will be up in the next doc build.

2. The way child process termination is implemented for each platform

...
should be documented.

Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added)

3. On most platforms there is a way to send a "quit" command to the process

...
and a way to kill it by force ("kill -9"). Clarifying what is happening in the default cases and having a way to terminate one way or the other (or a combination of both, that is requesting termination but force-kill after a time if it didn't die) On windows there is an issue with this because the "termination request" message can only be received if you didn't build a console application but a windows application (with a WinMain) but that only means that the message could be ignored by the process if it's console application (there are ways to have both WinMain and a console window, it's just a bit more clumsy to setup).

In our (Softbank Robotics EU) use of child process tracking we need both termination request and forced.

One could argue that the termination request do not need to be implemented by this library but it would be very useful to have a cross platform function doing this for simple cases.

From how I understand it, you send the terminate request do the HWND handle, not the proc handle. That technically means, that you cannot tell the process to exit, but a part of it. Hence I don't consider this the same as signaling the process to exit, which means it's not portable,

For the HWND handle I believe it is directly related to the process if it is a native windows application (a console application is similar but with additional code to instantiate the console window). All OS we use have a platform-specific way to request exit to a process (and signals are manageable on all these platforms). What I am suggesting is to provide a cross-platform function doing the platform-specific request, which would give a chance to the child process to kill itself. One can implement this with their own protocol too, but as platform-specific protocols are already there, having a ready way to do it would help.

...

thus I do argue it shouldn't be part of the library. If I have a child::request_exit function, it must work on all processes, not just some of the windows-procs.

Sending the request does work on all processes. Reading it is skipped by some special implementation of windows processes, like console processes. Even with a native windows process, or a linux process, you still have to do some platform-specific message processing in the child process code to manage this request, which is fine to me (and could also be generalized).

...

If there was a portable way to do it, it would be part of this library.

I don't see any way in which it is not possible to make it portable. I can provide example if you want? It's clumsy but it shows how we do it. (see below)

...

I think it's easy enough, you can get the pid (or even gid, if you use a group) and implement that in two lines. On windows you can get the HWND as an std::intptr_t through a pipe, though that's a bit more code.

Exactly. Which is why I think such function would be a good candidate for this library, although it's not a show stopper to me.

...

Thanks,

Klemens

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Klemens Morgenstern

11:02 a.m.

Am 06.06.2016 um 12:49 schrieb Klaim - Joël Lamotte:

...

On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...
So far I had time only to go quickly through the documentation and a bit in

...
the code so here are quick remarks:

1. It is not clear at all what happen when child object is destroyed.

Oh, ok that's only written in the reference atm. It terminates the child process if it wasn't detached or exited before. I added that, will be up in the next doc build.

2. The way child process termination is implemented for each platform

...
should be documented.

Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added)

3. On most platforms there is a way to send a "quit" command to the process

...
and a way to kill it by force ("kill -9"). Clarifying what is happening in the default cases and having a way to terminate one way or the other (or a combination of both, that is requesting termination but force-kill after a time if it didn't die) On windows there is an issue with this because the "termination request" message can only be received if you didn't build a console application but a windows application (with a WinMain) but that only means that the message could be ignored by the process if it's console application (there are ways to have both WinMain and a console window, it's just a bit more clumsy to setup).

In our (Softbank Robotics EU) use of child process tracking we need both termination request and forced.

One could argue that the termination request do not need to be implemented by this library but it would be very useful to have a cross platform function doing this for simple cases.

From how I understand it, you send the terminate request do the HWND handle, not the proc handle. That technically means, that you cannot tell the process to exit, but a part of it. Hence I don't consider this the same as signaling the process to exit, which means it's not portable,

For the HWND handle I believe it is directly related to the process if it is a native windows application (a console application is similar but with additional code to instantiate the console window). All OS we use have a platform-specific way to request exit to a process (and signals are manageable on all these platforms). What I am suggesting is to provide a cross-platform function doing the platform-specific request, which would give a chance to the child process to kill itself. One can implement this with their own protocol too, but as platform-specific protocols are already there, having a ready way to do it would help.

...
thus I do argue it shouldn't be part of the library. If I have a child::request_exit function, it must work on all processes, not just some of the windows-procs.

Sending the request does work on all processes. Reading it is skipped by some special implementation of windows processes, like console processes. Even with a native windows process, or a linux process, you still have to do some platform-specific message processing in the child process code to manage this request, which is fine to me (and could also be generalized).

And that's a bit problematic, since the same code might then behave differently on different platforms. But maybe it is possible to create some handling for the current process in the this_process namespace. E.g. a callback `this_process::on_exit_request(...)` or a bit `this_process::exit_requested()` so you can also skip the platform-specifics here.

...

...
If there was a portable way to do it, it would be part of this library.

I don't see any way in which it is not possible to make it portable. I can provide example if you want? It's clumsy but it shows how we do it. (see below) Please, that would help me very much. Also for the child process handling, regarding the this_process namespace.

...
I think it's easy enough, you can get the pid (or even gid, if you use a group) and implement that in two lines. On windows you can get the HWND as an std::intptr_t through a pipe, though that's a bit more code.

Exactly. Which is why I think such function would be a good candidate for this library, although it's not a show stopper to me. Well if there is a portable way, even if it's a hack, I'd look into adding it.

...

...
Thanks,

Klemens

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Klaim - Joël Lamotte

2:48 p.m.

On 6 June 2016 at 13:02, Klemens Morgenstern wrote:

...

Am 06.06.2016 um 12:49 schrieb Klaim - Joël Lamotte:

...
...
I don't see any way in which it is not possible to make it portable. I can provide example if you want? It's clumsy but it shows how we do it. (see below)

Please, that would help me very much. Also for the child process handling, regarding the this_process namespace.

...

...
...
group) and implement that in two lines. On windows you can get the HWND as an std::intptr_t through a pipe, though that's a bit more code.

Exactly. Which is why I think such function would be a good candidate for

I think it's easy enough, you can get the pid (or even gid, if you use a this library, although it's not a show stopper to me.

Well if there is a portable way, even if it's a hack, I'd look into adding it.

OK let me gather our open-source and closed source code in one example and I'll send it to you privately. Our solution is not perfect but it seems to work well. Joël Lamotte

Rob Stewart

7 Jun 7 Jun

8:37 a.m.

On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" wrote:

...

On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...
2. The way child process termination is implemented for each platform

...
should be documented.

Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added)

That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL. (You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.) On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period. Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be. ___ Rob (Sent from my portable computation engine)

Klemens Morgenstern

8:59 a.m.

Am 07.06.2016 um 10:37 schrieb Rob Stewart:

...

On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" wrote:

...
On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...
2. The way child process termination is implemented for each platform

...
should be documented.

Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added) That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.

(You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)

On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.

Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be. I actually thought the same thing, but it is an issue of security. Consider this:

ipstream is; child c("prog", std_in < is); is << generate_input() << endl; Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here. Regarding the WM_CLOSE version: I currently don't think that is a really portable solution, since you signal the HWND not the Process, which means that console programs will cause problems here. Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added. Now if that happens, I still will not change the behaviour in child, but you might be able to do this then: soft_exit se (child("thingy"), milliseconds(100)); It really depends on how this can be achieved on windows; I didn't look into that, because it's not part of the Process handling in the WinAPI, but maybe there's a workaround.

...

___ Rob

(Sent from my portable computation engine)

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Brendon Costa

11:21 a.m.

I am vaguely following this thread as I don't have lots of time but am interested in the process library. I have also written a C++ process (and signals) library supporting windows/OSX/Linux/Android (for an employer though so I can't propose it for boost). It actually suprised me how many platform differences existed that needed to be worked around in writing this library compared to others I have written. I also did something similar to Klemens if what I undertood is correct, I did a fallback scenario of send TERM if not closed within time period send a KILL. I didn't really like it though and wonder if the std::thread destructor behavior is better to std::terminate if the child is still running but not detached and treat it more like a bug to destroy an active process instance than something to rely on to close it. Not having a SIGTERM equivilant for windows processes bothered me as SIGKILL is just too nasty as a default as mentioned. So I looked into this quite thoroughly. For GUI processes you can easily send WM_CLOSE message to it to simulate something akin to a SIGTERM. I think I used a WM_CLOSE from memory. I know there was a choice between QUIT and CLOSE, I think CLOSE was more suitable but I cant recall the exact reason now and may be wrong. For console processes I only found bad options. I found something that worked quite well but ended up dropping it in the end as it was not required and was super hacky IMO. I only ever wanted to SIGTERM my own processes in the end so just used a non generic solution of a named pipe to listen for signals and raise() to local process. The GenerateConsoleCtrlEvent() comes close to doing what we want but its got the worst behaviour in how to choose who gets the signal delivered and is not sufficient (even when creating processes to detach/attach to other consoles to generate the events). In the end I followed the basic idea proposed at: http://www.latenighthacking.com/projects/2003/sendSignal/ With extra ideas proposed in comment (20110401) and a few extra changes of my own. I got this working well in all scenarios I could think of to test on win32 and 64 up to Win7 (I think I went down to XP but cant recall for sure). The idea is basically to use the mechanism that GenerateConsoleCtrlEvent() uses to deliver the signal, but you choose the process where to deliver it yourself instead of using the stupid rules that GenerateConsoleCtrlEvent() follows. The idea is to create a new "helper" process, have it register a signal handler, raise a signal to itself and in the handler walk back the stack to find the entry point where the signal was generated. It turns out that the function where the signal was generated (I think it was kernel32!CtrlRoutine) this function has the correct prototype for CeateRemoteThread and is in the kernel32.dll so is in the same location in all processes on the system and is how GenerateConsoleCtrlEvent() is implemented. So you basically then find the process you care about, create a remote thread in it to call the specified function and presto you end up with a SIGTERM that can be handled by a normal signal() handler in a console process. I hope my research was useful, but this approach was not really acceptable to me and I dont think would be to the boost community. Maybe another idea might arise from it though, in which case I would love to hear about it. On 7 June 2016 at 18:59, Klemens Morgenstern wrote:

...

Am 07.06.2016 um 10:37 schrieb Rob Stewart:

...
On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" wrote:

...
On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

2. The way child process termination is implemented for each platform

...
...
should be documented.

Makes sense. It's TerminateProcess on windows and "kill -9" on posix. (also added)

That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.

(You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)

On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.

Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be.

I actually thought the same thing, but it is an issue of security. Consider this:

ipstream is;

child c("prog", std_in < is); is << generate_input() << endl;

Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here.

Regarding the WM_CLOSE version: I currently don't think that is a really portable solution, since you signal the HWND not the Process, which means that console programs will cause problems here. Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added. Now if that happens, I still will not change the behaviour in child, but you might be able to do this then:

soft_exit se (child("thingy"), milliseconds(100));

It really depends on how this can be achieved on windows; I didn't look into that, because it's not part of the Process handling in the WinAPI, but maybe there's a workaround.

___

...
Rob

(Sent from my portable computation engine)

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Klemens Morgenstern

12:23 p.m.

Am 07.06.2016 um 13:21 schrieb Brendon Costa:

...

I am vaguely following this thread as I don't have lots of time but am interested in the process library.

I have also written a C++ process (and signals) library supporting windows/OSX/Linux/Android (for an employer though so I can't propose it for boost). It actually suprised me how many platform differences existed that needed to be worked around in writing this library compared to others I have written.

I also did something similar to Klemens if what I undertood is correct, I did a fallback scenario of send TERM if not closed within time period send a KILL. I didn't really like it though and wonder if the std::thread destructor behavior is better to std::terminate if the child is still running but not detached and treat it more like a bug to destroy an active process instance than something to rely on to close it.

Well, I don't give a timeout, it just terminates. That is, you have to explicitly call wait or join, or wait_for/wait_until if you want a timeout.

...

Not having a SIGTERM equivilant for windows processes bothered me as SIGKILL is just too nasty as a default as mentioned. So I looked into this quite thoroughly. For GUI processes you can easily send WM_CLOSE message to it to simulate something akin to a SIGTERM. I think I used a WM_CLOSE from memory. I know there was a choice between QUIT and CLOSE, I think CLOSE was more suitable but I cant recall the exact reason now and may be wrong.

For console processes I only found bad options. I found something that worked quite well but ended up dropping it in the end as it was not required and was super hacky IMO. I only ever wanted to SIGTERM my own processes in the end so just used a non generic solution of a named pipe to listen for signals and raise() to local process.

The GenerateConsoleCtrlEvent() comes close to doing what we want but its got the worst behaviour in how to choose who gets the signal delivered and is not sufficient (even when creating processes to detach/attach to other consoles to generate the events). In the end I followed the basic idea proposed at:

http://www.latenighthacking.com/projects/2003/sendSignal/

With extra ideas proposed in comment (20110401) and a few extra changes of my own. I got this working well in all scenarios I could think of to test on win32 and 64 up to Win7 (I think I went down to XP but cant recall for sure).

The idea is basically to use the mechanism that GenerateConsoleCtrlEvent() uses to deliver the signal, but you choose the process where to deliver it yourself instead of using the stupid rules that GenerateConsoleCtrlEvent() follows. Well, I could put each process in it's own group and just try to signal arbitrary values. But that would break the console and might not even work.

...

The idea is to create a new "helper" process, have it register a signal handler, raise a signal to itself and in the handler walk back the stack to find the entry point where the signal was generated. It turns out that the function where the signal was generated (I think it was kernel32!CtrlRoutine) this function has the correct prototype for CeateRemoteThread and is in the kernel32.dll so is in the same location in all processes on the system and is how GenerateConsoleCtrlEvent() is implemented.

So you basically then find the process you care about, create a remote thread in it to call the specified function and presto you end up with a SIGTERM that can be handled by a normal signal() handler in a console process.

I hope my research was useful, but this approach was not really acceptable to me and I dont think would be to the boost community. Maybe another idea might arise from it though, in which case I would love to hear about it.

It's definitely intertesting, thank you. Though you're unfortunately right, that's a bit too much hacking for a stable library.

Rob Stewart

8 Jun 8 Jun

2:06 a.m.

On June 7, 2016 4:59:59 AM EDT, Klemens Morgenstern wrote:

...

Am 07.06.2016 um 10:37 schrieb Rob Stewart:

...
On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" wrote:

...
On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...
It's TerminateProcess on windows and "kill -9" on posix.

...
That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.

(You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)

On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.

Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be. I actually thought the same thing, but it is an issue of security. Consider this:

ipstream is;

child c("prog", std_in < is); is << generate_input() << endl;

Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here.

I see your point. You could provide a default timeout for that case and include a function permitting the user to override that default.

...

Regarding the WM_CLOSE version: I currently don't think that is a really portable solution, since you signal the HWND not the Process, which means that console programs will cause problems here.

I mentioned that. Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().

...

Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added.

The remote thread technique sounds interesting.

...

Now if that happens, I still will not change the behaviour in child, but you might be able to do this then:

soft_exit se (child("thingy"), milliseconds(100));

Why not c.terminate(milliseconds(100)) and c.kill()? ___ Rob (Sent from my portable computation engine)

Klemens Morgenstern

7:05 a.m.

Am 08.06.2016 um 04:06 schrieb Rob Stewart:

...

On June 7, 2016 4:59:59 AM EDT, Klemens Morgenstern wrote:

...
Am 07.06.2016 um 10:37 schrieb Rob Stewart:

...
On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" wrote:

...
On 6 June 2016 at 12:00, Klemens Morgenstern wrote:

...
It's TerminateProcess on windows and "kill -9" on posix.

...
That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.

(You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)

On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.

Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be. I actually thought the same thing, but it is an issue of security. Consider this:

ipstream is;

child c("prog", std_in < is); is << generate_input() << endl;

Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here.

I see your point. You could provide a default timeout for that case and include a function permitting the user to override that default.

Sry, but no. I don't see the point of timeouts, especially default ones. In this case it is quite clear, that there's no point in continuing, so why put in timeout?

...

...
Regarding the WM_CLOSE version: I currently don't think that is a really portable solution, since you signal the HWND not the Process, which means that console programs will cause problems here.

I mentioned that. Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().

Difference is: every posix program can catch SIGTERM, not every windows program will get WM_CLOSE - console applications will never receive that. It depends on the compile options. Now for this to be part of the process core library (i.e. included in boost/process.hpp) it would need to work like this both platforms: //father child c(...); c.request_exit(); //child this_process::on_exit_request(std::function func); That's not possible, so it won't be in the library. It might be added to the platform-extensions though, i.e. you could have something like the following functions if you include boost/process/posix.hpp or boost/process/windows.hpp (needs to be distinguished by #ifdef) posix::send_terminate(child &); windows::send_wm_close(child&); windows::send_console_kill(child&); I have no problem having a platform-specific function there, but if I have a function in the multi-platform part of boost.process it has to behave the same everywhere.

...

...
Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added.

The remote thread technique sounds interesting.

...
Now if that happens, I still will not change the behaviour in child, but you might be able to do this then:

soft_exit se (child("thingy"), milliseconds(100));

Why not c.terminate(milliseconds(100)) and c.kill()?

Yeah would be possible, too. Though as written above, there won't be a terminate-request member-function of child. But what you could do is this. some_magic_terminate_request(c); //your SIGTERM impl. if (!c.wait_for(milliseconds(100)) c.terminate();

...

___ Rob

(Sent from my portable computation engine)

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Rob Stewart

8:33 a.m.

On June 8, 2016 3:05:37 AM EDT, Klemens Morgenstern wrote:

...

Am 08.06.2016 um 04:06 schrieb Rob Stewart:

...
...
Consider this:

ipstream is; child c("prog", std_in < is); is << generate_input() << endl;

Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here.

I see your point. You could provide a default timeout for that case and include a function permitting the user to override that default.

Sry, but no. I don't see the point of timeouts, especially default ones. In this case it is quite clear, that there's no point in continuing, so why put in timeout?

Your approach gives the child no chance to shut down gracefully. It cannot close database connections, remove temporary files, clean up content in shared memory, whatever.

...

...
Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().

Difference is: every posix program can catch SIGTERM, not every windows program will get WM_CLOSE - console applications will never receive that. It depends on the compile options.

Some console apps can be written to process such messages, but most won't, of course. What's more, if an app doesn't handle the message, you're no worse off than when you just call TerminateProcess().

...

Now for this to be part of the process core library (i.e. included in boost/process.hpp) it would need to work like this both platforms:

//father child c(...); c.request_exit();

//child this_process::on_exit_request(std::function func);

That's not possible, so it won't be in the library. It might be added to the platform-extensions though, i.e. you could have something like the following functions if you include boost/process/posix.hpp or boost/process/windows.hpp (needs to be distinguished by #ifdef)

posix::send_terminate(child &); windows::send_wm_close(child&); windows::send_console_kill(child&);

I have no problem having a platform-specific function there, but if I have a function in the multi-platform part of boost.process it has to behave the same everywhere.

I see that you only want to send the nice termination signal if the child can install a handler in a common way. I was missing that side of the equation. The Windows issue is whether it's possible to determine, at compile time or runtime, whether an association has a GUI message loop or not. If so, you'd have to install a message loop handler for WM_CLOSE, and if not, you'd have to use SetConsoleCtrlHandler() to install a callback. I don't know much about Windows message handler loops, but you could install a message hook if there isn't a better way to integrate into the app's own message loop.

...

...
...
Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added.

The remote thread technique sounds interesting.

...
Now if that happens, I still will not change the behaviour in child, but you might be able to do this then:

soft_exit se (child("thingy"), milliseconds(100));

Why not c.terminate(milliseconds(100)) and c.kill()?

Yeah would be possible, too. Though as written above, there won't be a terminate-request member-function of child. But what you could do is this.

some_magic_terminate_request(c); //your SIGTERM impl.

if (!c.wait_for(milliseconds(100)) c.terminate();

I'm confused why you think some_magic_terminate_request() should be a free function, while terminate() is a member function. ___ Rob (Sent from my portable computation engine)

Klemens Morgenstern

9:19 a.m.

Am 08.06.2016 um 10:33 schrieb Rob Stewart:

...

On June 8, 2016 3:05:37 AM EDT, Klemens Morgenstern wrote:

...
Am 08.06.2016 um 04:06 schrieb Rob Stewart:

...
...
Consider this:

ipstream is; child c("prog", std_in < is); is << generate_input() << endl;

Not if generate_input throws, I need to terminate the child, elsewise I get a deadlock (since "prog" waits for input). If you do not want this, you can detach it or join it. In that it is similar to std::thread, though it doesn't terminate the current process. Now using a timeout would be possible, but I really don't like to set an arbitrary value here.

I see your point. You could provide a default timeout for that case and include a function permitting the user to override that default.

Sry, but no. I don't see the point of timeouts, especially default ones. In this case it is quite clear, that there's no point in continuing, so why put in timeout?

Your approach gives the child no chance to shut down gracefully. It cannot close database connections, remove temporary files, clean up content in shared memory, whatever.

A timeout doesn't make sense if you have no standard way to signal the child to exit. If you implement your version of that, you can use chlid::wait_for and then terminate. The child class is meant to be joined before destructing; the terminate on destruction is for exceptions.

...

...
...
Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().

Difference is: every posix program can catch SIGTERM, not every windows program will get WM_CLOSE - console applications will never receive that. It depends on the compile options.

Some console apps can be written to process such messages, but most won't, of course. What's more, if an app doesn't handle the message, you're no worse off than when you just call TerminateProcess().

...
Now for this to be part of the process core library (i.e. included in boost/process.hpp) it would need to work like this both platforms:

//father child c(...); c.request_exit();

//child this_process::on_exit_request(std::function func);

That's not possible, so it won't be in the library. It might be added to the platform-extensions though, i.e. you could have something like the following functions if you include boost/process/posix.hpp or boost/process/windows.hpp (needs to be distinguished by #ifdef)

posix::send_terminate(child &); windows::send_wm_close(child&); windows::send_console_kill(child&);

I have no problem having a platform-specific function there, but if I have a function in the multi-platform part of boost.process it has to behave the same everywhere.

I see that you only want to send the nice termination signal if the child can install a handler in a common way. I was missing that side of the equation. The Windows issue is whether it's possible to determine, at compile time or runtime, whether an association has a GUI message loop or not. If so, you'd have to install a message loop handler for WM_CLOSE, and if not, you'd have to use SetConsoleCtrlHandler() to install a callback.

That, AND it ought to be the default on the system for exit requests.

...

I don't know much about Windows message handler loops, but you could install a message hook if there isn't a better way to integrate into the app's own message loop.

It doesn't work really, especially since SetConsoleCtrlHandler does only allow two values (Ctrl+C & Ctrl+Break), which are both not clearly termination requests, but more of a terminate command I think. Also to do that, I would need to put all those child processes on a new process group, which would mean, that Ctrl+C will NOT be transmitted to them. That's also a dealbreaker.

...

...
...
...
Joël Lamotte recommended a similar solution and will send me some examples, so a child::request_exit() function might be added.

The remote thread technique sounds interesting.

...
Now if that happens, I still will not change the behaviour in child, but you might be able to do this then:

soft_exit se (child("thingy"), milliseconds(100));

Why not c.terminate(milliseconds(100)) and c.kill()?

Yeah would be possible, too. Though as written above, there won't be a terminate-request member-function of child. But what you could do is this.

some_magic_terminate_request(c); //your SIGTERM impl.

if (!c.wait_for(milliseconds(100)) c.terminate();

I'm confused why you think some_magic_terminate_request() should be a free function, while terminate() is a member function.

Because I would be fine to have it as an extension in the posix- or windows-namespace, but not in the core-library. Hence it won't be a method of child.

...

___ Rob

(Sent from my portable computation engine)

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Gavin Lambert

9 Jun 9 Jun

2:57 a.m.

On 8/06/2016 21:19, Klemens Morgenstern wrote:

...

It doesn't work really, especially since SetConsoleCtrlHandler does only allow two values (Ctrl+C & Ctrl+Break), which are both not clearly termination requests, but more of a terminate command I think. Also to do that, I would need to put all those child processes on a new process group, which would mean, that Ctrl+C will NOT be transmitted to them. That's also a dealbreaker.

Ctrl+C is equivalent to SIGINT and Ctrl+Break is equivalent to SIGBREAK. So if you want to send one of those two signals then GenerateConsoleCtrlEvent is probably the right way to go. (You might need to use CreateRemoteThread to make the child raise it itself rather than doing it in the parent process though, which requires different versions for 32-bit vs. 64-bit processes.) If you want to raise SIGTERM, this is possible but you'd have to use CreateRemoteThread to call raise(SIGTERM) directly in the child process. Where this gets a little tricky is that you'd need to call the right version of "raise", which differs depending on the CRT the child process uses (static or dynamic, different compiler versions, etc) -- and the required code differs between 32-bit and 64-bit processes too. And of course both of the above are dependent on the console app developer actually intercepting and processing them as a graceful shutdown -- otherwise it's little different from a TerminateProcess. Services are really the only kind of executable that have a defined way to externally start/stop them programmatically. Other process types have interactions that *can* do so but they're intended for user action rather than programmatic action. (Even WM_CLOSE for GUI apps isn't always the right answer, depending on app design.) Typically graceful programmatic shutdown is only possible when both parent and child are made by the same developers so that they can do something custom to request it.

Klemens Morgenstern

7:05 a.m.

Am 09.06.2016 um 04:57 schrieb Gavin Lambert:

...

On 8/06/2016 21:19, Klemens Morgenstern wrote:

...
It doesn't work really, especially since SetConsoleCtrlHandler does only allow two values (Ctrl+C & Ctrl+Break), which are both not clearly termination requests, but more of a terminate command I think. Also to do that, I would need to put all those child processes on a new process group, which would mean, that Ctrl+C will NOT be transmitted to them. That's also a dealbreaker.

Ctrl+C is equivalent to SIGINT and Ctrl+Break is equivalent to SIGBREAK. So if you want to send one of those two signals then GenerateConsoleCtrlEvent is probably the right way to go. (You might need to use CreateRemoteThread to make the child raise it itself rather than doing it in the parent process though, which requires different versions for 32-bit vs. 64-bit processes.)

If you want to raise SIGTERM, this is possible but you'd have to use CreateRemoteThread to call raise(SIGTERM) directly in the child process. Where this gets a little tricky is that you'd need to call the right version of "raise", which differs depending on the CRT the child process uses (static or dynamic, different compiler versions, etc) -- and the required code differs between 32-bit and 64-bit processes too. But I would need to know which binary is executed, then use LoadLibrary to get it, and then look for the entry point, so I can pass it to CreateRemoveThread, right? Do you have an example on how to do this? I would need to be able to this for ANY process having the raise function.

And of course both of the above are dependent on the console app developer actually intercepting and processing them as a graceful shutdown -- otherwise it's little different from a TerminateProcess.

Services are really the only kind of executable that have a defined way to externally start/stop them programmatically. Other process types have interactions that *can* do so but they're intended for user action rather than programmatic action. (Even WM_CLOSE for GUI apps isn't always the right answer, depending on app design.)

Typically graceful programmatic shutdown is only possible when both parent and child are made by the same developers so that they can do something custom to request it.

Sad, but true. That's why it probably won't be part of boost.process.

...

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Gavin Lambert

8:26 a.m.

On 9/06/2016 19:05, Klemens Morgenstern wrote:

...

Am 09.06.2016 um 04:57 schrieb Gavin Lambert:

...
On 8/06/2016 21:19, Klemens Morgenstern wrote:

...
It doesn't work really, especially since SetConsoleCtrlHandler does only allow two values (Ctrl+C & Ctrl+Break), which are both not clearly termination requests, but more of a terminate command I think. Also to do that, I would need to put all those child processes on a new process group, which would mean, that Ctrl+C will NOT be transmitted to them. That's also a dealbreaker.

Ctrl+C is equivalent to SIGINT and Ctrl+Break is equivalent to SIGBREAK. So if you want to send one of those two signals then GenerateConsoleCtrlEvent is probably the right way to go. (You might need to use CreateRemoteThread to make the child raise it itself rather than doing it in the parent process though, which requires different versions for 32-bit vs. 64-bit processes.)

If you want to raise SIGTERM, this is possible but you'd have to use CreateRemoteThread to call raise(SIGTERM) directly in the child process. Where this gets a little tricky is that you'd need to call the right version of "raise", which differs depending on the CRT the child process uses (static or dynamic, different compiler versions, etc) -- and the required code differs between 32-bit and 64-bit processes too. But I would need to know which binary is executed, then use LoadLibrary to get it, and then look for the entry point, so I can pass it to CreateRemoveThread, right? Do you have an example on how to do this? I would need to be able to this for ANY process having the raise function.

No, I don't. I did find some convoluted code that will do SIGINT/SIGBREAK (it does SIGBREAK) at https://github.com/walware/statet/blob/master/de.walware.statet.r.console.co... (link sharing for the lulz), but I don't think this is ASLR safe as it assumes that the address of the kernel32 console control handler is the same in both processes. The version that uses AttachConsole is probably safer instead. (Not that SIGINT/SIGBREAK would be a good idea for Boost.Process anyway, since many console apps -- eg. cmd itself -- handle that as an interactive command and might display an "are you sure" prompt or similar. Which is also just like WM_CLOSE, incidentally.) Other signals are more problematic, because signals aren't actually a thing on Windows, and so the others are just emulated in the CRT. (And processes not raising them themselves probably aren't expecting them anyway.)

...

...
Typically graceful programmatic shutdown is only possible when both parent and child are made by the same developers so that they can do something custom to request it.

Sad, but true. That's why it probably won't be part of boost.process.

Now, if you can guarantee that both parent and child use Boost.Process, you could probably do it (eg. use a pipe or other IPC mechanism to let the parent trigger the child calling raise(SIGTERM) on itself.) I'm not convinced it'd be worth the effort though.

Klemens Morgenstern

4:39 p.m.

Am 09.06.2016 um 10:26 schrieb Gavin Lambert:

...

On 9/06/2016 19:05, Klemens Morgenstern wrote:

...
Am 09.06.2016 um 04:57 schrieb Gavin Lambert:

...
On 8/06/2016 21:19, Klemens Morgenstern wrote:

...
It doesn't work really, especially since SetConsoleCtrlHandler does only allow two values (Ctrl+C & Ctrl+Break), which are both not clearly termination requests, but more of a terminate command I think. Also to do that, I would need to put all those child processes on a new process group, which would mean, that Ctrl+C will NOT be transmitted to them. That's also a dealbreaker.

Ctrl+C is equivalent to SIGINT and Ctrl+Break is equivalent to SIGBREAK. So if you want to send one of those two signals then GenerateConsoleCtrlEvent is probably the right way to go. (You might need to use CreateRemoteThread to make the child raise it itself rather than doing it in the parent process though, which requires different versions for 32-bit vs. 64-bit processes.)

If you want to raise SIGTERM, this is possible but you'd have to use CreateRemoteThread to call raise(SIGTERM) directly in the child process. Where this gets a little tricky is that you'd need to call the right version of "raise", which differs depending on the CRT the child process uses (static or dynamic, different compiler versions, etc) -- and the required code differs between 32-bit and 64-bit processes too. But I would need to know which binary is executed, then use LoadLibrary to get it, and then look for the entry point, so I can pass it to CreateRemoveThread, right? Do you have an example on how to do this? I would need to be able to this for ANY process having the raise function.

No, I don't. I did find some convoluted code that will do SIGINT/SIGBREAK (it does SIGBREAK) at https://github.com/walware/statet/blob/master/de.walware.statet.r.console.co... (link sharing for the lulz), but I don't think this is ASLR safe as it assumes that the address of the kernel32 console control handler is the same in both processes. The version that uses AttachConsole is probably safer instead.

Well, it keeps the style of the WinAPI :) "The Windows API has done more to retard skill development than anything since COBOL maintenance." (Larry O'Brien)

...

(Not that SIGINT/SIGBREAK would be a good idea for Boost.Process anyway, since many console apps -- eg. cmd itself -- handle that as an interactive command and might display an "are you sure" prompt or similar. Which is also just like WM_CLOSE, incidentally.)

Other signals are more problematic, because signals aren't actually a thing on Windows, and so the others are just emulated in the CRT. (And processes not raising them themselves probably aren't expecting them anyway.)

I came accross boost.application, that would probably be a better candidate to emulate signals if we need a custom receiver on the child-side.

...

...
...
Typically graceful programmatic shutdown is only possible when both parent and child are made by the same developers so that they can do something custom to request it.

Sad, but true. That's why it probably won't be part of boost.process.

Now, if you can guarantee that both parent and child use Boost.Process, you could probably do it (eg. use a pipe or other IPC mechanism to let the parent trigger the child calling raise(SIGTERM) on itself.) I'm not convinced it'd be worth the effort though.

Neighter am I, I'd do it, if it was easy. However: I've decided to definitely not add this until boost.process is reviewed - which will still take a lot of work. We have not discussed four approaches to this problem and no one cuts it in my opinion. If one shall be added, it must be tested before. So I'd accept a pull request if it is either marked experimental or a platform-extension. There will however not be a standard way to do this until the first release of the library. I don't think having no SIGTERM will be a deal-breaker and if we have an accepted boost.process it will be much easier to experiment with that stuff. I will however summarize the ideas and reopen the discussion when I think boost.process is ready for experimental features.

...

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Oswin Krause

8 Jun 8 Jun

8:32 p.m.

...

That's not possible, so it won't be in the library. It might be added to the platform-extensions though, i.e. you could have something like the following functions if you include boost/process/posix.hpp or boost/process/windows.hpp (needs to be distinguished by #ifdef) I only partially followed the discussion. I think the starting point of

Hi, this was that the library needs a way to kill a child for example to prevent a deadlock when a pipe breaks, etc. What would in this case be the default behaviour on linux? I would think that a solution would be perfectly fine if it behaved as expected on each operating system, even if the behaviour is different for the different operating systems.

Klemens Morgenstern

9:42 p.m.

Am 08.06.2016 um 22:32 schrieb Oswin Krause:

...

Hi,

...
That's not possible, so it won't be in the library. It might be added to the platform-extensions though, i.e. you could have something like the following functions if you include boost/process/posix.hpp or boost/process/windows.hpp (needs to be distinguished by #ifdef) I only partially followed the discussion. I think the starting point of this was that the library needs a way to kill a child for example to prevent a deadlock when a pipe breaks, etc. What would in this case be the default behaviour on linux? I would think that a solution would be perfectly fine if it behaved as expected on each operating system, even if the behaviour is different for the different operating systems.

Well not, that's actually settled. The question is, what happes per default on constructor call of the child-handle. And that's terminate, i.e. kill, without any timeout. The question is if we can have a SIGTERM equivalent on windows, and it doesn't seem like it. There is a WM_QUIT message, but that's not send to the process, but its HWNDs, rendering it useless, if you have a console app.

...

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Damien Buhl

16 Jun 16 Jun

2:26 p.m.

On 04/06/2016 21:52, Klemens Morgenstern wrote:

...

Hi all,

the boost.process 0.6 library is now in alpha, that is: it has some (compiling) examples, a documentation and tests that pass on windows as well as on linux.

It basically allows you to manage a process like this:

pstream pipe;

child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in < "data.csv", std_out > pipe, std_err > null, start_dir="./bin");

std::string data; pipe >> data;

c.wait(); cout << c.exit_code() << endl;

You can find the documentation here: http://klemens-morgenstern.github.io/process/ Get the source code from here: https://github.com/klemens-morgenstern/boost-process

If you want to use it on windows, you currently would need to clone my boost.winapi fork also: https://github.com/klemens-morgenstern/winapi

As for the scope of the library, it is meant to implement a small, portable layer to allow platform-independent process management. Thereby it's functionality is (unless I missed something) basically what is possible on posix as well as on windows. There are some platform extension, but I would not recommend to use them. It is also possible (though not yet documented) to write extensions, and if they work on both platforms I will gladly add them.

It is however not the goal of the library, to be the be-all and end-all of process management; in past disscussions, different ideas where brought up, like implementing a DSEL for that or only allowing asynchronous communication. This library will never be that, since it would be much more fitting to build this on-top of it. E.g. a DSEL could be written (e.g. boost.shell) using boost.process underneath, so the library could concentrate only on the DSEL.

It doesn't depend on boost.iostreams anymore, though it still uses boost.asio, boost.fusion and boost.filesystem.

As for the open issues: some error handling is still lacking (but that'd be something like being unable to duplicate a handle), which would be required for proper extension, which is also why it's not documented. Also there's still an open discussion about the naming of the class and functions (child, system, spawn), so that may change also. But I consider this simple enough for a easy refactoring. If anyone has naming ideas, please tell me. One Idea was 'subprocess' but 'process::subprocess' just looks too strange to me.

I hope you try it out, and give me some feedbacks. Or write angry error reports, both helps.

Thanks!

Klemens

Hi Klemens, We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function. The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux. Without the use of vfork, you end up in situations where you get the following error: `boost::process::detail::posix_start: fork(2) failed: Cannot allocate memory` I think changing from fork to vfork is not much and brings alot of advantages, but one must be aware that at_fork handler won't be called. But this is not important as fork is used to do execve afterwards in Boost.Process. Cheers, -- Damien Buhl alias daminetreg

Klemens Morgenstern

4:02 p.m.

...

Hi Klemens,

We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function.

The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux.

Without the use of vfork, you end up in situations where you get the following error: `boost::process::detail::posix_start: fork(2) failed: Cannot allocate memory`

I think changing from fork to vfork is not much and brings alot of advantages, but one must be aware that at_fork handler won't be called. But this is not important as fork is used to do execve afterwards in Boost.Process. Hi Damien,

I appreciate your problem, but I am not sure it's that easy. There's on major problem: vfork is removed from the posix standard as of POSIX.1-2008. And I'm trying to conform to posix, not linux. So I guess, I wouldn´t use vfork as default, but it might be possible to add a property, which will cause the library to use that. I.e. you write: boost::process::child c("java.exe", "overhead.jar", boost::process::posix::use_vfork); But I'd need to be able to check if vfork is available, so I can disable the property if not. Would that be sufficient for your problem? Btw.: if you have any particular experience, that might help me improve things, please share it. I'm always interested in that; the current state is just based on how I'd use it. Thanks, Klemens

Damien Buhl

17 Jun 17 Jun

8:36 a.m.

On 16/06/2016 18:02, Klemens Morgenstern wrote:

...

...
Hi Klemens,

We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function.

The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux.

Without the use of vfork, you end up in situations where you get the following error: `boost::process::detail::posix_start: fork(2) failed: Cannot allocate memory`

I think changing from fork to vfork is not much and brings alot of advantages, but one must be aware that at_fork handler won't be called. But this is not important as fork is used to do execve afterwards in Boost.Process. Hi Damien,

I appreciate your problem, but I am not sure it's that easy. There's on major problem: vfork is removed from the posix standard as of POSIX.1-2008. And I'm trying to conform to posix, not linux.

So I guess, I wouldn´t use vfork as default, but it might be possible to add a property, which will cause the library to use that. I.e. you write:

boost::process::child c("java.exe", "overhead.jar", boost::process::posix::use_vfork);

But I'd need to be able to check if vfork is available, so I can disable the property if not.

Would that be sufficient for your problem?

Naturally if the code still compiles on windows even though I'm using boost::process::posix::use_vfork(_if_possible) then yes for me all will be fine. But on the other hand from a library design point of view, shouldn't the library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me.

...

On 17/06/2016 01:21, Gavin Lambert wrote:

...
Is this specifically for NOMMU linux?

MMU architectures shouldn't have this issue, as fork does a shared-memory-copy-on-write mapping so that the pages aren't actually duplicated unless written to, and the subsequent exec* unmaps the pages so this never happens (other than a bit of stack).

NOMMU architectures don't support those kinds of mappings, so have to be rewritten to use vfork instead (which is generally unsafe unless immediately followed by exec*, unless you know what you're doing).

Although the last time that I played with NOMMU, fork used to just fail; perhaps it's been changed to make copies of all the pages instead? If so, that would indeed be problematic for large parent processes.

Hi Gavin ;)

It's with an MMU but the parent process on this product is a memory-hungry-monster-jvm and the fork fails. I can be wrong, and it looks like you know more from what's happening there, but as long as I can remember the issue arose due to virtual memory commit, because fork on linux even though is optimized to not copy the whole process pages but to do copy-on-write, still needs to commit virtual memory, and overcommitment is not always allowed, or by default is heuristically allowed. Quoting man proc, the default is 0:

...

/proc/sys/vm/overcommit_memory This file contains the kernel virtual memory accounting mode. Values are:

0: heuristic overcommit (this is the default) 1: always overcommit, never check 2: always check, never overcommit

So it might fail, and it happens to fail often on a 80% RSS usage of the device RAM from the parent process which will do the fork. While vfork doesn't need to commit vm memory, or at least not so much as the parent process, but just what's needed for pid_t and a call to execve I find it a better alternative when someone does only do a fork followed by execve. But there might be considerations I don't understand, I never worked on the linux kernel process creation code, and so I'm just sharing from a dumb user point-of-view. Perhaps we could just remove the jvm, but as I proposed this to the java team they didn't invited me for the coffee anymore. :p

Klemens Morgenstern

9:45 a.m.

Am 17.06.2016 um 10:36 schrieb Damien Buhl:

...

...
...
Hi Klemens,

We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function.

The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux.

Without the use of vfork, you end up in situations where you get the following error: `boost::process::detail::posix_start: fork(2) failed: Cannot allocate memory`

I think changing from fork to vfork is not much and brings alot of advantages, but one must be aware that at_fork handler won't be called. But this is not important as fork is used to do execve afterwards in Boost.Process. Hi Damien,

I appreciate your problem, but I am not sure it's that easy. There's on major problem: vfork is removed from the posix standard as of POSIX.1-2008. And I'm trying to conform to posix, not linux.

So I guess, I wouldn´t use vfork as default, but it might be possible to add a property, which will cause the library to use that. I.e. you write:

boost::process::child c("java.exe", "overhead.jar", boost::process::posix::use_vfork);

But I'd need to be able to check if vfork is available, so I can disable the property if not.

Would that be sufficient for your problem? Naturally if the code still compiles on windows even though I'm using boost::process::posix::use_vfork(_if_possible) then yes for me all will be fine. But on the other hand from a library design point of view, shouldn't the

On 16/06/2016 18:02, Klemens Morgenstern wrote: library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me. Again: obsolete and now removed in the posix-standard and that's what I ought to go with. Please note, that I'm trying to provide two platforms: Posix & Windows. Not Linux & Windows. Thereby I want to provide the most common way for both platforms; and though I really appreciate your scenario, I would not consider it the common way.

...

...
On 17/06/2016 01:21, Gavin Lambert wrote:

...
Is this specifically for NOMMU linux?

MMU architectures shouldn't have this issue, as fork does a shared-memory-copy-on-write mapping so that the pages aren't actually duplicated unless written to, and the subsequent exec* unmaps the pages so this never happens (other than a bit of stack).

NOMMU architectures don't support those kinds of mappings, so have to be rewritten to use vfork instead (which is generally unsafe unless immediately followed by exec*, unless you know what you're doing).

Although the last time that I played with NOMMU, fork used to just fail; perhaps it's been changed to make copies of all the pages instead? If so, that would indeed be problematic for large parent processes. Hi Gavin ;)

It's with an MMU but the parent process on this product is a memory-hungry-monster-jvm and the fork fails. It seems a bit strange to me, that you would use boost.process here, instead of - you know - java.io.process. But I guess you want

...

I can be wrong, and it looks like you know more from what's happening there, but as long as I can remember the issue arose due to virtual memory commit, because fork on linux even though is optimized to not copy the whole process pages but to do copy-on-write, still needs to commit virtual memory, and overcommitment is not always allowed, or by default is heuristically allowed. That's probably right, because elsewise you'd get an error in the forked

I think you underestimate the dangers of vfork; because you know, two processes sharing memory (including the stack!) can be rather fun. There's a reason it was removed and so it will be optional in boost.process. It would be a platform extension, so no, this would NOT compile with windows. I thought about turning the platform extensions into NOPs , but that's just too weird - especially since things like signal(SIGCHLD, ...) are also provided. There you should rather use the preprocessor and put an #ifdef there - then it's obvious what you're doing. Also I'd need a #define to know whether vfork is available, that I would probably provide. So you could then write something like that: child c("ls" #if defined(BOOST_POSIX_HAS_VFORK) , posix::use_vfork #endif ); I guess I can check that via 'CLONE_VFORK'. performance and your java-developers want a job, so JNI is the way to go? process and there would be no way of getting to know it.

Damien Buhl

1:31 p.m.

On 17/06/2016 11:45, Klemens Morgenstern wrote:

...

...
...
So I guess, I wouldn´t use vfork as default, but it might be possible to add a property, which will cause the library to use that. I.e. you write:

boost::process::child c("java.exe", "overhead.jar", boost::process::posix::use_vfork);

But I'd need to be able to check if vfork is available, so I can disable the property if not.

Would that be sufficient for your problem? Naturally if the code still compiles on windows even though I'm using boost::process::posix::use_vfork(_if_possible) then yes for me all will be fine. But on the other hand from a library design point of view, shouldn't the library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me. Again: obsolete and now removed in the posix-standard and that's what I ought to go with. Please note, that I'm trying to provide two

[snip] platforms: Posix & Windows. Not Linux & Windows. Thereby I want to provide the most common way for both platforms; and though I really appreciate your scenario, I would not consider it the common way.

I think you underestimate the dangers of vfork; because you know, two processes sharing memory (including the stack!) can be rather fun. There's a reason it was removed and so it will be optional in boost.process.

It would be a platform extension, so no, this would NOT compile with windows. I thought about turning the platform extensions into NOPs , but that's just too weird - especially since things like signal(SIGCHLD, ...) are also provided. There you should rather use the preprocessor and put an #ifdef there - then it's obvious what you're doing. Also I'd need a #define to know whether vfork is available, that I would probably provide. So you could then write something like that:

child c("ls" #if defined(BOOST_POSIX_HAS_VFORK) , posix::use_vfork #endif );

I guess I can check that via 'CLONE_VFORK'. Yes that's fine, I just find it better when a library user don't have to use #ifdef at all, that's why I was asking. But that's all the difficulty of Boost.Process : it's pretty hard to abstract something which is really different in the little detail.

Just for reference, this is what posix_spawn does in the glibc, which is roughly equivalent to what Boost.Process provides via the child class, they are also feared about using vfork, so you are surely right to avoid it as much as possible : In glibc /sysdeps/posix/spawni.c

...

/* Generate the new process. */ if ((flags & POSIX_SPAWN_USEVFORK) != 0 /* If no major work is done, allow using vfork. Note that we might perform the path searching. But this would be done by a call to execvp(), too, and such a call must be OK according to POSIX. */ || ((flags & (POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF | POSIX_SPAWN_SETSCHEDPARAM | POSIX_SPAWN_SETSCHEDULER | POSIX_SPAWN_SETPGROUP | POSIX_SPAWN_RESETIDS)) == 0 && file_actions == NULL)) new_pid = __vfork (); else new_pid = __fork ();

The rationale of posix_spawn here is a good addition to what we discuss I think : http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#ta... And as your goal is Posix + Windows, I understand you don't want to deal with specialities of linux / solaris / ... so I think you're right it's safer the default to be fork.

...

...
[snip]

It's with an MMU but the parent process on this product is a memory-hungry-monster-jvm and the fork fails. It seems a bit strange to me, that you would use boost.process here, instead of - you know - java.io.process. But I guess you want performance and your java-developers want a job, so JNI is the way to go?

It's off-topic but to explain: The fact is that they just reuse a C++ library we had for long, where we added a JNI interface for it, which is launching at one point processes that are not developed internally and that we cannot implement as library because we don't have the source code. Additionally I believe that in the version of jvm implementation used (oracle - arm - early 1.7) I think java.io.process does fork+exec, which has been changed after because of similar issues: http://bugs.java.com/view_bug.do?bug_id=6868160 Thanks for dealing with my wishes to vfork :)

Klemens Morgenstern

10:54 p.m.

...

The rationale of posix_spawn here is a good addition to what we discuss I think : http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#ta...

And as your goal is Posix + Windows, I understand you don't want to deal with specialities of linux / solaris / ... so I think you're right it's safer the default to be fork.

...
...
[snip]

It's with an MMU but the parent process on this product is a memory-hungry-monster-jvm and the fork fails. It seems a bit strange to me, that you would use boost.process here, instead of - you know - java.io.process. But I guess you want performance and your java-developers want a job, so JNI is the way to go?

It's off-topic but to explain: The fact is that they just reuse a C++ library we had for long, where we added a JNI interface for it, which is launching at one point processes that are not developed internally and that we cannot implement as library because we don't have the source code.

Additionally I believe that in the version of jvm implementation used (oracle - arm - early 1.7) I think java.io.process does fork+exec, which has been changed after because of similar issues: http://bugs.java.com/view_bug.do?bug_id=6868160

Thanks for dealing with my wishes to vfork :)

Alright, the feature is in now, you find that in boost/process/posix.hpp - it's not documented though. If you pass posix::use_vfork to the function it will work. BUT: I just removed the test for that, because it ran locally (on my debian VM) but travis didn't like it very much - for some unknown reason. Maybe you have some experience and might be able to find a fix. I'd be really happy to have a test for this - I'd just need travis to work.

Damien Buhl

21 Jun 21 Jun

6:03 a.m.

On 18/06/2016 00:54, Klemens Morgenstern wrote:

...

Alright, the feature is in now, you find that in boost/process/posix.hpp - it's not documented though. If you pass posix::use_vfork to the function it will work.

BUT: I just removed the test for that, because it ran locally (on my debian VM) but travis didn't like it very much - for some unknown reason. Maybe you have some experience and might be able to find a fix. I'd be really happy to have a test for this - I'd just need travis to work.

Thanks I'll give it a try and see if I it's possible to have it run on travis-ci.

Gavin Lambert

12:14 a.m.

On 17/06/2016 21:45, Klemens Morgenstern wrote:

...

...
But on the other hand from a library design point of view, shouldn't the library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me.

Again: obsolete and now removed in the posix-standard and that's what I ought to go with. Please note, that I'm trying to provide two platforms: Posix & Windows. Not Linux & Windows. Thereby I want to provide the most common way for both platforms; and though I really appreciate your scenario, I would not consider it the common way.

Possibly of interest is that according to the notes vfork+exec (and fork+exec, for that matter) was deprecated in favour of posix_spawn. Which appears to be less well documented, but does exist in some environments at least.

Damien Buhl

6:05 a.m.

On 21/06/2016 02:14, Gavin Lambert wrote:

...

On 17/06/2016 21:45, Klemens Morgenstern wrote:

...
...
But on the other hand from a library design point of view, shouldn't the library have the best smart default in terms of performance and overhead on a given platform ? Instead of having a flag telling : please do it the same but efficiently ? Because on linux vfork is nothing but obsoleted and for a scenario of using execve looks better to me.

Again: obsolete and now removed in the posix-standard and that's what I ought to go with. Please note, that I'm trying to provide two platforms: Posix & Windows. Not Linux & Windows. Thereby I want to provide the most common way for both platforms; and though I really appreciate your scenario, I would not consider it the common way.

Possibly of interest is that according to the notes vfork+exec (and fork+exec, for that matter) was deprecated in favour of posix_spawn. Which appears to be less well documented, but does exist in some environments at least.

Yes it's the code I quoted from the glibc, ( basically Boost.Process is another posix_spawn, brought to another level though) :

...

The rationale of posix_spawn here is a good addition to what we discuss I think : http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#ta...

In glibc /sysdeps/posix/spawni.c implementation of posix_spawn:

...

/* Generate the new process. */ if ((flags & POSIX_SPAWN_USEVFORK) != 0 /* If no major work is done, allow using vfork. Note that we might perform the path searching. But this would be done by a call to execvp(), too, and such a call must be OK according to POSIX. */ || ((flags & (POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF | POSIX_SPAWN_SETSCHEDPARAM | POSIX_SPAWN_SETSCHEDULER | POSIX_SPAWN_SETPGROUP | POSIX_SPAWN_RESETIDS)) == 0 && file_actions == NULL)) new_pid = __vfork (); else new_pid = __fork ();

Gavin Lambert

16 Jun 16 Jun

11:21 p.m.

On 17/06/2016 02:26, Damien Buhl wrote:

...

We have been using the Boost.Process state from Boris Schäling for more than 3 year in productive use on linux embedded devices, and one thing that we had to change is to use vfork instead of the fork function.

The problem with the fork + execve is that it produce a RAM copy of the process page, while vfork+execve just don't. And when the parent process launching child is pretty big, imagine a jvm using some jni library using Boost.Process... it duplicates the whole Jvm im RAM to then deallocate all at the moment of execve. vfork doesn't have this issue on linux.

Is this specifically for NOMMU linux? MMU architectures shouldn't have this issue, as fork does a shared-memory-copy-on-write mapping so that the pages aren't actually duplicated unless written to, and the subsequent exec* unmaps the pages so this never happens (other than a bit of stack). NOMMU architectures don't support those kinds of mappings, so have to be rewritten to use vfork instead (which is generally unsafe unless immediately followed by exec*, unless you know what you're doing). Although the last time that I played with NOMMU, fork used to just fail; perhaps it's been changed to make copies of all the pages instead? If so, that would indeed be problematic for large parent processes.

3082

Age (days ago)

3099

Last active (days ago)

List overview

Download

32 comments

8 participants

participants (8)

Brendon Costa
Damien Buhl
Edward Diener
Gavin Lambert
Klaim - Joël Lamotte
Klemens Morgenstern
Oswin Krause
Rob Stewart