Stonewall Ballard wrote:
I think I found the cause of this problem. It seems that the caller of interrupt_all should be holding the mutex associated with the condition on which the threads are waiting.
This gave me the clue to try that: http://www.opengroup.org/onlinepubs/009695399/functions/pthread_cond_broadca...
The pthread_cond_broadcast() or pthread_cond_signal() functions may be called by a thread whether or not it currently owns the mutex that threads calling pthread_cond_wait() or pthread_cond_timedwait() have associated with the condition variable during their waits; however, if predictable scheduling behavior is required, then that mutex shall be locked by the thread calling pthread_cond_broadcast() or pthread_cond_signal().
thread::interrupt() calls pthread_cond_broadcast in pthread/thread.cpp.
Although "predictable scheduling" doesn't seem like it should include a failure to wake up, taking the mutex around the call to thread_pool::interrupt_all() appears to be 100% reliable.
I can patch my app to do that, but I don't think there's a general solution. The documentation should include a note that thread::interrupt() isn't reliable unless the caller is holding the mutex associated with the condition variable on which the interrupted thread is waiting.
Of course, this could be a bug in the OS X pthreads implementation as well.
Hi, FWIW, I ran that test of yours several times with varying parameters on my machine (quad core, 64bit, linux) and it did not show a single failure. Of course, since it is not a deterministic effect even on your machine, failure to reproduce does not really mean much, but well, I thought you might like to hear anyway :-) And I totally agree: Predictable scheduling should not be required to wake up all threads, especially since the document also says <snip> The pthread_cond_broadcast() or pthread_cond_signal() functions may be called by a thread whether or not it currently owns the mutex [...] </cite> As for boiling down the application for others to inspect: Your debugger showed that the thread is still in wait() after the interrupt call. Can you assure that ALL worker threads are in wait() prior to the interrupt? * If yes: There seems to be no connection with the interlocked queue, the sleep and so on. It should be possible to get rid of all that for a much simpler test program * If no: OK, there seems to be a connection between the wait(), the interrupt and the sleep and/or mutex. In any case, I would assume that by analysing the situation right before the interrupt, you should be able to reproduce the problem with much less code. Hope that helps in any way? Regards, Roland
- Stoney
I've discovered that under circumstances apparently related to timing and load, sending interrupt_all to a thread_group when all the threads are waiting on a boost::condition_variable leaves one thread waiting about 1/3 of the time. This is with boost 1_40_0 running on Mac OS X 10.6.2, with 32-bit boost libraries. Boost uses the posix thread system here. I boiled my app down to some test code that runs as a command-line app. It's a bit longer than I'd like, but this configuration seems to be necessary to invoke the problem. The test uses a queue to pass "tasks" from the main thread to worker threads, and another queue to pass "results" back to the main thread. The problem is most apparent when all the tasks are finished and the queue empties, so that all the worker threads are waiting on the input queue when the main thread sends interrupt_all.
I've looked at the waiting thread in a debugger when this happens, and found that it has been interrupted, but is still waiting on the condition. It looks like it just got missed by the interrupt_all. This is more likely to happen when there are a lot of worker threads (16, or one per core in my testing).
The test code is parked at http://sb.org/ThreadTest.zip, 20KB. It's an XCode 3.2 project, but the five source files could be readily compiled and run in any Unix environment.
I don't see any errors in the code that could cause these failures. There is a work-around, which is to interrupt the waiting thread again. This required a modified version of thread_group so I could do a timed_join_all on it.
I welcome any suggestions about what could be wrong here, or ways to simplify the test to make it more suitable for a bug report.
- Stoney