[asio][coroutine] forbid_suspend blocks
I've implemented a dumb linter using Python + clang API that allows you to mark certain blocks as “atomic in respect to coroutines” using C++11 attributes. Usage example follows: void f() { int yield; g(yield); [[bt::forbid_suspend]] { // Comment this call to make linter accept the code g(yield); t(); } } You can change bt::forbid_suspend to another identifier by tuning settings on the beginning of the script. The prefix forbid was inspired by Rust's linters[1] as you cannot disallow the linter from inside the forbid_suspend block. Some limitations follow: - A suspend point is identified by looking references to any id equals to yield in the last argument passed to function calls. It means that yield[ignored_ec] and alike will be correctly identified, but it needs to be named yield and it needs to be the last argument to the function call. - I haven't found a way to extract attributes related to COMPOUND_STMT nodes with clang API, so I manually look for token gaps and do a simple regex. It means you cannot use macros to [[bt::forbid_suspend]] as the macro won't be expanded. - You better call the script with '-x c++' and also ' -I/usr/lib/clang/5.0.1/include'. Like I've said in the beginning of the email, it's a *dumb* linter, which means it doesn't try to handle all cases. Therefore, I don't hope to do many changes to it or maintain it to work against newer releases of clang and I'm only uploading it to gist (not fullblow Github): < https://gist.github.com/vinipsmaker2/d930fbe5b7597432b021effe618da171>. It should also be easy to hack as it is very small. I'm releasing the linter under public domain. Enjoy. I have been using fibers in a daily basis on the last months of my job to solve networking problems on a gateway project. Some of the technologies I use are: - Boost.Asio and its spawn function. - A few custom synchronization primitives for the fibers (e.g. mutex, semaphore). - Eventually I'll have to write a custom spawn function that returns a joinable handle so I can kill the usage of a fiber::barrier in a few places. I've been using the linter for about a week already and looks fine. My use case are: - To prevent iterator invalidation in regions accessing shared variables. - To guarantee events order in algorithms that handle races (cannot yield while task YYY hasn't finished). The project I use it in is not an open source project, so I cannot talk much more about it. Also, I don't claim my solution is original. I wouldn't be surprised if the very same solution exists in the wild (please let me know if you know of a similar approach). This is not a new problem. It is know for some time. For instance, in P0171, Gor Nishanov noted this same problem: “In coroutines the suspend point is clearly marked with await, which tells the reader that something unusual happens in this function and allows the reader, for example, to confirm whether the lifetimes of the objects of interest align with the lifetime of the coroutine or not, whether some locks need to be acquired to protect some concurrently accessed data, and whether some locks need to be released before the execution reaches the suspend point.” — http://open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0171r0.html, 2015 However, the solution he presented was to manually examine whole blocks of code without help from the compiler. I argue that the solution he presented is as error-prone as manually looking for exit paths that don't free acquired resources in languages lacking RAII. And I've been spoiled too much by RAII. Furthermore, I take this comment of his with highly suspicious look: “Since we are on the subject of maintenance nightmares, we would like to offer a conjecture that the absence of the await in P0114 is a likely source of many maintenance nightmares. Without a syntactic marker to signal to the person reading the code that something funny is going on, it is impossible to tell whether the following code is correct or not” — http://open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0171r0.html If I had to give a vote to the proposal he was defending or Chris' proposal, I'd probably have chosen Chris'. [1] https://doc.rust-lang.org/reference/attributes.html# lint-check-attributes -- Vinícius dos Santos Oliveira https://vinipsmaker.github.io/
On Wed, Aug 15, 2018 at 5:06 AM Vinícius dos Santos Oliveira via Boost
the absence of the await in P0114 is a likely source of many maintenance nightmares
We have no idea how P0114 would play out because no one took the time to implement it. We only heard from the opposing counsel (Gor), and he only offered opinions. I spearheaded an effort to implement P0114 in clang and we made some progress, including necessary modifications to the original paper to make things smooth and also address some issues raised by other stakeholders. But the talent I was working with did not have enough time available to complete the project. One of the modifications was an "opt-in" model of resumable expressions, to avoid the problem where code unsuspectingly becomes suspended at a place that should not suspend (for example, while holding a mutex). High quality engineers familiar with Clang/LLVM are mostly already working on projects, and also employed by large corporations. The mere existence of Coroutines TS, plus the assumption of correctness conferred by the author's employer, discourages significant investment in alternate implementations. Thanks
Thanks for the info. I forgot to acknowledge last time. Em qua, 15 de ago de 2018 às 23:11, Vinnie Falco via Boost < boost@lists.boost.org> escreveu:
On Wed, Aug 15, 2018 at 5:06 AM Vinícius dos Santos Oliveira via Boost
wrote: the absence of the await in P0114 is a likely source of many maintenance nightmares
We have no idea how P0114 would play out because no one took the time to implement it. We only heard from the opposing counsel (Gor), and he only offered opinions. I spearheaded an effort to implement P0114 in clang and we made some progress, including necessary modifications to the original paper to make things smooth and also address some issues raised by other stakeholders. But the talent I was working with did not have enough time available to complete the project. One of the modifications was an "opt-in" model of resumable expressions, to avoid the problem where code unsuspectingly becomes suspended at a place that should not suspend (for example, while holding a mutex).
High quality engineers familiar with Clang/LLVM are mostly already working on projects, and also employed by large corporations. The mere existence of Coroutines TS, plus the assumption of correctness conferred by the author's employer, discourages significant investment in alternate implementations.
Thanks
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- Vinícius dos Santos Oliveira https://vinipsmaker.github.io/
Just to share a report on experience with this approach, Em qua, 15 de ago de 2018 às 09:06, Vinícius dos Santos Oliveira < vini.ipsmaker@gmail.com> escreveu:
[...] allows you to mark certain blocks as “atomic in respect to coroutines” using C++11 attributes.
Usage example follows:
void f() { int yield; g(yield); [[bt::forbid_suspend]] { // Comment this call to make linter accept the code g(yield); t(); } }
Well, I've stumbled on limitations for this solution as I had to re-enable
suspensions on the middle of a block. Something like:
std::visit([&](auto& e) {
using T = std::decay_t
P0171, Gor Nishanov noted this same problem:
“In coroutines the suspend point is clearly marked with await, which tells the reader that something unusual happens in this function and allows the reader, for example, to confirm whether the lifetimes of the objects of interest align with the lifetime of the coroutine or not, whether some locks need to be acquired to protect some concurrently accessed data, and whether some locks need to be released before the execution reaches the suspend point.” — http://open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0171r0.html, 2015
I expanded my previous thoughts on this argument on the assert_exclusive_strand_ref<T> documentation. “Since we are on the subject of maintenance nightmares, we would like to
offer a conjecture that the absence of the await in P0114 is a likely source of many maintenance nightmares. Without a syntactic marker to signal to the person reading the code that something funny is going on, it is impossible to tell whether the following code is correct or not” — http://open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0171r0.html
Can be done. Just like the compiler helps with static_assert, it can help with some static_assert_{forbid,allow}_suspend() and these asserts can in turn be used to build abstractions just like the assert_exclusive_strand_ref<T>. I've converted a few scary code sections from my job to use assert_exclusive_strand_ref<T> instead relying on manually inspecting every this_fiber usage throughout the code. From personal experience: - The this_fiber argument in the end of every suspending function worked as a hinted mark. - Manually looking for suspension points to analyze function behaviours acted precisely as a maintenance nightmare. I've lost hours on very small blocks of code. - Relying on alternative approaches — the forbid suspend block that I've abandoned and the new alternative — gave me much more confidence on the codes as (1) the code became less brittle and (2) it was easier to analyze the code. [1] this_fiber completion token has a similar “mark” effect as the await keyword -- Vinícius dos Santos Oliveira https://vinipsmaker.github.io/
participants (2)
-
Vinnie Falco
-
Vinícius dos Santos Oliveira