CI Testing and failures
Sometimes CI testing fails because of a bug in a compiler. It is possible to designate in a jam file that a particular test should not be done for a particular compiler version so that CI testing succeeds, and therefore a PR can be more easily merged because the CI tests "succeed". But then the fact that the compiler has a bug is masked by such a change to the jam file. Is there a solution to such a problem, or maybe this is really not a problem at all ? What do others think about this ?
On 2020-04-26 18:17, Edward Diener via Boost wrote:
Sometimes CI testing fails because of a bug in a compiler. It is possible to designate in a jam file that a particular test should not be done for a particular compiler version so that CI testing succeeds, and therefore a PR can be more easily merged because the CI tests "succeed". But then the fact that the compiler has a bug is masked by such a change to the jam file. Is there a solution to such a problem, or maybe this is really not a problem at all ? What do others think about this ?
If the bug affects the library, and you still intend to support it, it follows that you have to add a workaround for the bug in the library. The failing test indicates that users of this compiler have an actual problem. If you don't intend to support this compile, just remove it from the CI. In the old build matrix this can't be done, except to disable it in the test/Jamfile.
On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:
On 2020-04-26 18:17, Edward Diener via Boost wrote:
Sometimes CI testing fails because of a bug in a compiler. It is possible to designate in a jam file that a particular test should not be done for a particular compiler version so that CI testing succeeds, and therefore a PR can be more easily merged because the CI tests "succeed". But then the fact that the compiler has a bug is masked by such a change to the jam file. Is there a solution to such a problem, or maybe this is really not a problem at all ? What do others think about this ?
If the bug affects the library, and you still intend to support it, it follows that you have to add a workaround for the bug in the library. The failing test indicates that users of this compiler have an actual problem.
So if there is no workaround, should the feature for the particular compiler/version be disabled ? Should the test "succeed" for the feature for the particular compiler/version by either being disabled in Boost Build or by "passing" because the feature is disabled for the compiler/version ?
If you don't intend to support this compile, just remove it from the CI. In the old build matrix this can't be done, except to disable it in the test/Jamfile.
I do not think the entire compiler/version should be removed from testing if just a very small part of the library does not work because of a bug. What you are saying is that the tests are meant to succeed rather than show a valid bug in some compiler/version. The opposite viewpoint is that a failing test is a good indication of a compiler/version bug and therefore a failing test is not an indication of a failing library. For the purposes of Github changes, particularly PRs, with CI tests, it does seem as if the tests should succeed, which would back your viewpoint.
On 2020-04-27 19:52, Edward Diener via Boost wrote:
On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:
On 2020-04-26 18:17, Edward Diener via Boost wrote:
Sometimes CI testing fails because of a bug in a compiler. It is possible to designate in a jam file that a particular test should not be done for a particular compiler version so that CI testing succeeds, and therefore a PR can be more easily merged because the CI tests "succeed". But then the fact that the compiler has a bug is masked by such a change to the jam file. Is there a solution to such a problem, or maybe this is really not a problem at all ? What do others think about this ?
If the bug affects the library, and you still intend to support it, it follows that you have to add a workaround for the bug in the library. The failing test indicates that users of this compiler have an actual problem.
So if there is no workaround, should the feature for the particular compiler/version be disabled ? Should the test "succeed" for the feature for the particular compiler/version by either being disabled in Boost Build or by "passing" because the feature is disabled for the compiler/version ?
If you don't intend to support this compile, just remove it from the CI. In the old build matrix this can't be done, except to disable it in the test/Jamfile.
I do not think the entire compiler/version should be removed from testing if just a very small part of the library does not work because of a bug. What you are saying is that the tests are meant to succeed rather than show a valid bug in some compiler/version. The opposite viewpoint is that a failing test is a good indication of a compiler/version bug and therefore a failing test is not an indication of a failing library. For the purposes of Github changes, particularly PRs, with CI tests, it does seem as if the tests should succeed, which would back your viewpoint.
That is, of course, for you to decide. If only a small part of the library is not supported on the particular compiler while the rest is still usable, you can document this limitation and exclude the tests of that part from the CI (e.g. via preprocessor checks or by factoring out the test to a separate executable and disabling it in the Jamfile). My main point is that if you want to deal with the compiler bug, you should find a workaround and keep the tests running. Otherwise, there's no point in running tests for the unsupported code on that compiler.
participants (2)
-
Andrey Semashev -
Edward Diener