[gil][test] HOWTO set up the Boost testing process for projects with extra prerequisites ?
Hi, I just merged some work to GIL's `develop` branch. This notably involves improvements to the testing infrastructure. Testing GIL is a bit involved, as it requires various backend libraries (PNG, JPEG, TIFF, etc.) to be installed. While the stand-alone testing works fine (https://github.com/boostorg/gil/blob/develop/.travis.yml#L304-L306), the integrated testing will thus fail, until the testing logic is augmented by the necessary instructions to a) install the prerequisites and b) invoke the build system with the appropriate flags (https://github.com/boostorg/gil/blob/develop/io/test/Jamfile#L13-L22) Thus my question: do we have any documentation that explains how to do this ? What do I (as GIL maintainer) or any testers need to do for the integrated testing to work ? (I can't resist: all these libraries that have special requirements really shouldn't be built and tested in such a monolithic process. It's a burden to all tester, maintainers, and packagers. It would be far better to handle them more independently.) Stefan -- ...ich hab' noch einen Koffer in Berlin...
AMDG On 11/17/2017 09:29 AM, Stefan Seefeld via Boost wrote:
I just merged some work to GIL's `develop` branch. This notably involves improvements to the testing infrastructure. Testing GIL is a bit involved, as it requires various backend libraries (PNG, JPEG, TIFF, etc.) to be installed. While the stand-alone testing works fine (https://github.com/boostorg/gil/blob/develop/.travis.yml#L304-L306), the integrated testing will thus fail, until the testing logic is augmented by the necessary instructions to a) install the prerequisites and b) invoke the build system with the appropriate flags (https://github.com/boostorg/gil/blob/develop/io/test/Jamfile#L13-L22)
Please use the Boost.Build modules for these libraries instead. You can make the tests run only when the corresponding libraries are available, like this: using zlib ; run png_test.cpp : # args : # input-files : # requirements [ ac.check-library /zlib//zlib : <library>/zlib//zlib : <build>no ] # in English, if the library is found, use it, otherwise, # skip this test ;
Thus my question: do we have any documentation that explains how to do this ? What do I (as GIL maintainer) or any testers need to do for the integrated testing to work ?
(I can't resist: all these libraries that have special requirements really shouldn't be built and tested in such a monolithic process. It's a burden to all tester, maintainers, and packagers. It would be far better to handle them more independently.)
In Christ, Steven Watanabe
On 17.11.2017 11:42, Steven Watanabe via Boost wrote:
AMDG
On 11/17/2017 09:29 AM, Stefan Seefeld via Boost wrote:
I just merged some work to GIL's `develop` branch. This notably involves improvements to the testing infrastructure. Testing GIL is a bit involved, as it requires various backend libraries (PNG, JPEG, TIFF, etc.) to be installed. While the stand-alone testing works fine (https://github.com/boostorg/gil/blob/develop/.travis.yml#L304-L306), the integrated testing will thus fail, until the testing logic is augmented by the necessary instructions to a) install the prerequisites and b) invoke the build system with the appropriate flags (https://github.com/boostorg/gil/blob/develop/io/test/Jamfile#L13-L22)
Please use the Boost.Build modules for these libraries instead. You can make the tests run only when the corresponding libraries are available, like this:
using zlib ;
That would be https://github.com/boostorg/build/blob/develop/src/tools/zlib.jam, yes ?
run png_test.cpp : # args : # input-files : # requirements [ ac.check-library /zlib//zlib : <library>/zlib//zlib : <build>no ] # in English, if the library is found, use it, otherwise, # skip this test ;
So not only are you opposed to the idea to modularize Boost's build logic, you also want to compile all prerequisites in the same monolithic bundle ? That's insane ! Or have I misunderstood what you are proposing ? Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Fri, Nov 17, 2017 at 10:51 AM, Stefan Seefeld via Boost < boost@lists.boost.org> wrote:
Or have I misunderstood what you are proposing ?
^^ That.. https://github.com/boostorg/build/blob/develop/src/tools/zlib.jam#L73 -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net
On 17.11.2017 11:59, Rene Rivera via Boost wrote:
On Fri, Nov 17, 2017 at 10:51 AM, Stefan Seefeld via Boost < boost@lists.boost.org> wrote:
Or have I misunderstood what you are proposing ?
^^ That..
https://github.com/boostorg/build/blob/develop/src/tools/zlib.jam#L73
OK, much better. But that doesn't answer the original question: How does a tester need to set up his system (and configure the build) for the GIL IO tests to work ? In either case does he a) need to install additional libraries b) add instructions (no matter whether it's a `-s...` command line option or an entry in the `user-config.jam` file) to locate them. So with our current process we need to make all testers aware of that, or else the GIL tests will fail. (And no, I don't consider testing with in-process-built-prereq libs an option, as that doesn't represent the world that our users live in). It's this existing practice that leads to a lot of noise in the test matrix (at least for the Boost.Python tests), as there are a ton of test runs that simply aren't set up correctly, and thus making it hard to identifying which of the many test failures are significant and should be acted upon by developers, rather than ignored. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Fri, Nov 17, 2017 at 11:10 AM, Stefan Seefeld via Boost < boost@lists.boost.org> wrote:
On 17.11.2017 11:59, Rene Rivera via Boost wrote:
On Fri, Nov 17, 2017 at 10:51 AM, Stefan Seefeld via Boost < boost@lists.boost.org> wrote:
Or have I misunderstood what you are proposing ?
^^ That..
https://github.com/boostorg/build/blob/develop/src/tools/zlib.jam#L73
OK, much better. But that doesn't answer the original question: How does a tester need to set up his system (and configure the build) for the GIL IO tests to work ?
In either case does he
a) need to install additional libraries
Probably.
b) add instructions (no matter whether it's a `-s...` command line option or an entry in the `user-config.jam` file) to locate them.
Like in my other email.. Add to their user-config.jam where/how to get those libraries.
So with our current process we need to make all testers aware of that, or else the GIL tests will fail.
Right.
(And no, I don't consider testing with in-process-built-prereq libs an option, as that doesn't represent the world that our users live in).
It represents some percentage of users. For example game devs build just about all the libraries they depend on. I suspect so does the trading field. And anything that is embedded or close to it. But I understand what you meant when you said "users" ;-)
It's this existing practice that leads to a lot of noise in the test matrix (at least for the Boost.Python tests), as there are a ton of test runs that simply aren't set up correctly, and thus making it hard to identifying which of the many test failures are significant and should be acted upon by developers, rather than ignored.
Couldn't tell you if that's true or not. As I haven't looked at other devs libraries. All I can say is that it's my responsibility to make sure my library gets tested appropriately. Which includes bugging the testers for appropriate setups and fixes. And making sure anything I depend on is available and/or that I can adjust to it being there. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net
AMDG On 11/17/2017 10:10 AM, Stefan Seefeld via Boost wrote:
On 17.11.2017 11:59, Rene Rivera via Boost wrote:
On Fri, Nov 17, 2017 at 10:51 AM, Stefan Seefeld via Boost < boost@lists.boost.org> wrote:
Or have I misunderstood what you are proposing ?
^^ That..
https://github.com/boostorg/build/blob/develop/src/tools/zlib.jam#L73
OK, much better. But that doesn't answer the original question: How does a tester need to set up his system (and configure the build) for the GIL IO tests to work ? In either case does he
a) need to install additional libraries b) add instructions (no matter whether it's a `-s...` command line option or an entry in the `user-config.jam` file) to locate them.
Right. We should probably have a list of additional dependencies here: http://www.boost.org/development/running_regression_tests.html
So with our current process we need to make all testers aware of that, or else the GIL tests will fail. (And no, I don't consider testing with in-process-built-prereq libs an option, as that doesn't represent the world that our users live in).
Building the prerequisite libraries is not really any different, as it just moves the problem to finding the sources.
It's this existing practice that leads to a lot of noise in the test matrix (at least for the Boost.Python tests), as there are a ton of test runs that simply aren't set up correctly, and thus making it hard to identifying which of the many test failures are significant and should be acted upon by developers, rather than ignored.
The method I described won't generate as much noise, as the tests are skipped instead of failing if the library can't be found. In Christ, Steven Watanabe
On Fri, Nov 17, 2017 at 10:42 AM, Steven Watanabe via Boost < boost@lists.boost.org> wrote:
AMDG
On 11/17/2017 09:29 AM, Stefan Seefeld via Boost wrote:
I just merged some work to GIL's `develop` branch. This notably involves improvements to the testing infrastructure. Testing GIL is a bit involved, as it requires various backend libraries (PNG, JPEG, TIFF, etc.) to be installed. While the stand-alone testing works fine (https://github.com/boostorg/gil/blob/develop/.travis.yml#L304-L306), the integrated testing will thus fail, until the testing logic is augmented by the necessary instructions to a) install the prerequisites and b) invoke the build system with the appropriate flags (https://github.com/boostorg/gil/blob/develop/io/test/Jamfile#L13-L22)
Please use the Boost.Build modules for these libraries instead. You can make the tests run only when the corresponding libraries are available, like this:
using zlib ;
run png_test.cpp : # args : # input-files : # requirements [ ac.check-library /zlib//zlib : <library>/zlib//zlib : <build>no ] # in English, if the library is found, use it, otherwise, # skip this test ;
Although ideally it should be: import zlib ; run ... ; And then ask the testers to add the appropriate "using zlib ... ;" to their configuration. And that should also include any CI testing you do also. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net
participants (3)
-
Rene Rivera -
Stefan Seefeld -
Steven Watanabe