Attention library authors.. Library requirements testing is now active.
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes: 1. Most libraries are now tested automatically if they have a usable test directory. There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones those are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I plan to blindly remove that custom list of extra test targets. 2. Most structure library requirements are now checked both in the regular full testing and in the CI testing. I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>. At the moment it reports 17 libraries as having errors. There are also some warnings, i.e. not requirement violations but just some stuff that may be wrong. The script also runs as part of the general full testing and any problems are reported in each library's results as a single entry at the top of the tables. Here's how one of the test failures looks like < http://tinyurl.com/zbe9gal> [1] [2]. It would be great if library authors address those errors in some form ASAP. I know that for some of those reported libraries there will need to be some discussions as to how to best address the errors. Feel free to start additional threads for those as needed. [1] < http://www.boost.org/development/tests/develop/developer/output/teeks99-09f-...
[2] Note there are currently some false failures in this aspect from a build system "bug". The bug is already fixed but tests will take some time to cycle to reflect the bug fix. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Hi Rene, can you point me to an active resource for defining how test directories are meant to be implemented in boost? Thanks, Matt On 15/06/2016 3:14 a.m., Rene Rivera wrote:
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones those are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I plan to blindly remove that custom list of extra test targets.
2. Most structure library requirements are now checked both in the regular full testing and in the CI testing.
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>. At the moment it reports 17 libraries as having errors. There are also some warnings, i.e. not requirement violations but just some stuff that may be wrong. The script also runs as part of the general full testing and any problems are reported in each library's results as a single entry at the top of the tables. Here's how one of the test failures looks like < http://tinyurl.com/zbe9gal> [1] [2].
It would be great if library authors address those errors in some form ASAP. I know that for some of those reported libraries there will need to be some discussions as to how to best address the errors. Feel free to start additional threads for those as needed.
[1] < http://www.boost.org/development/tests/develop/developer/output/teeks99-09f-...
[2] Note there are currently some false failures in this aspect from a build system "bug". The bug is already fixed but tests will take some time to cycle to reflect the bug fix.
On Tue, Jun 14, 2016 at 6:26 PM, Soul Studios
Hi Rene, can you point me to an active resource for defining how test directories are meant to be implemented in boost?
The integration requirements has a minimal example Jamfile < http://www.boost.org/development/requirements.html#Integration>. If you need something more than that you can ask in the list and also look at what other libraries do. Note that those integration requirements list aliases that are currently not used by testing. As they are meant for future implementation of the test systems. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Rene Rivera
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
How will this interact with libraries that already have their own CI setup? Is your CI triggered whenever a submodule is synced to boostorg/boost? If so, there's going to be quite a bit of redundant testing being done, and since all of boostorg is sharing the same Travis resources (I think), that could quickly become a problem. Is this a valid concern? Louis
On Tue, Jun 14, 2016 at 9:16 PM, Louis Dionne
Rene Rivera
writes: As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now
implemented
some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
How will this interact with libraries that already have their own CI setup?
Is your CI triggered whenever a submodule is synced to boostorg/boost?
If so, there's going to be quite a bit of redundant testing being done, and since all of boostorg is sharing the same Travis resources (I think), that could quickly become a problem. Is this a valid concern?
My plan is to make how a library uses CI up to the authors. I'm working on a script that will hopefully make it really simple to have a library use CI testing and that authors can essentially drop in (it's the script I use to test Predef by the way). But I'm not going to force anyone to use it. If you already have CI testing you might not want to switch. But hopefully many authors will use it as it will handle multiple CI systems, multiple toolsets, etc. As for sharing resources.. I do believe they are shared across all Boost repos. But we wont be limited to Travis. And there's the possibility of paying for extra resources. And eventually I hope we can turn our existing custom testers into a CI pool also. Most of this the plans are not in the near future. The near ones are trying to clean up and reorganize libraries and test scripts to make it possible to figure out what we can do. And also to make doing releases as automatic and painless as possible for everyone. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Rene Rivera
[...]
My plan is to make how a library uses CI up to the authors. I'm working on a script that will hopefully make it really simple to have a library use CI testing and that authors can essentially drop in (it's the script I use to test Predef by the way). But I'm not going to force anyone to use it.
Cool. As long as Hana is not tested twice for every commit that's merged into master, which would be prohibitively expensive in terms of CI resources, that's fine. I'll see if it makes sense for Hana to switch to the CI scripts you provide. Thanks, Louis
On Tue, Jun 14, 2016 at 9:45 PM, Rene Rivera
On Tue, Jun 14, 2016 at 9:16 PM, Louis Dionne
wrote: Rene Rivera
writes: Is your CI triggered whenever a submodule is synced to boostorg/boost?
If so, there's going to be quite a bit of redundant testing being done, and since all of boostorg is sharing the same Travis resources (I think), that could quickly become a problem. Is this a valid concern?
As for sharing resources.. I do believe they are shared across all Boost repos. But we wont be limited to Travis. And there's the possibility of paying for extra resources. And eventually I hope we can turn our existing custom testers into a CI pool also.
This was discussed a bit last year when I was presenting my proposal for testing resources for the current system to the steering committee. I believe the general feeling was that they would love to see CI to the point where we need more Travis (or other) resources, and would entertain proposals once that happened. Tom
On Wednesday, 15 June 2016 11:52:03 MSK Rene Rivera wrote:
2. Most structure library requirements are now checked both in the regular full testing and in the CI testing.
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>.
In that report: libs/winapi: error: file not found; Did not find [project-root]/index.html file. The file is required for all libraries. Redirection to HTML documentation. libs/winapi: error: directory not found; Missing [project-root]/doc directory. The [project-root]/doc directory is required for all libraries. Sources to build with and built documentation for the library. If the library needs to build documentation from non-HTML files this location must be buildable with Boost Build. Boost.WinAPI does not have documentation, it's not a public library. What should I do about these errors? libs/winapi: error: file not found; Did not find [project-root]/include/boost/ [library].h* file. A single header for the library is required at [project- root]/include/boost/[library].h* if the library does not have a header directory at [project-root]/include/boost/[library]. I didn't know this was a requirement. Anyway, there's no single header for Boost.WinAPI and its headers are placed under boost/detail/winapi. Again, because it's not a public library and because it used to be part of Boost.Detail. libs/winapi: error: directory not found; Missing [project-root]/meta directory. The [project-root]/meta directory is required for all libraries. I thought meta/* is supposed to be created by someone else (you?) with a pull request. I'm not sure what I should do about it. Also there's a python error at the end of the report: Failed: 'b2' '--debug-configuration' '-j2' '-d0' '--check-libs-only' ERROR = 1 # /home/travis/build/boostorg/boost/status> 'b2' '--debug-configuration' '-j2' '-d0' '--check-libs-only' execution time 9.05490994453 seconds Traceback (most recent call last): File "/home/travis/build/boostorg/boost/../script.py", line 421, in <module> script_travis() File "/home/travis/build/boostorg/boost/../script.py", line 377, in __init__ commit=os.getenv("TRAVIS_COMMIT")) File "/home/travis/build/boostorg/boost/../script.py", line 248, in __init__ self.main() File "/home/travis/build/boostorg/boost/../script.py", line 296, in main getattr(self,action_m)() File "/home/travis/build/boostorg/boost/../script.py", line 391, in command_script self.command_base_build() File "/home/travis/build/boostorg/boost/../script.py", line 282, in command_base_build self.b2("-d0","--check-libs-only") File "/home/travis/build/boostorg/boost/../script.py", line 308, in b2 return utils.check_call(*cmd) File "/home/travis/build/boostorg/boost/../script.py", line 46, in check_call raise(SystemCallError([cwd].extend(command), result)) __main__.SystemCallError The command "python "${TRAVIS_BUILD_DIR}/../script.py" script" exited with 1. after_failure
On Wed, Jun 15, 2016 at 4:03 AM, Andrey Semashev
On Wednesday, 15 June 2016 11:52:03 MSK Rene Rivera wrote:
2. Most structure library requirements are now checked both in the
regular
full testing and in the CI testing.
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>.
In that report:
libs/winapi: error: file not found; Did not find [project-root]/index.html file. The file is required for all libraries. Redirection to HTML documentation. libs/winapi: error: directory not found; Missing [project-root]/doc directory. The [project-root]/doc directory is required for all libraries. Sources to build with and built documentation for the library. If the library needs to build documentation from non-HTML files this location must be buildable with Boost Build.
Boost.WinAPI does not have documentation, it's not a public library. What should I do about these errors?
I would add a single doc file saying that it's a "private" Boost library.
libs/winapi: error: file not found; Did not find [project-root]/include/boost/ [library].h* file. A single header for the library is required at [project- root]/include/boost/[library].h* if the library does not have a header directory at [project-root]/include/boost/[library].
I didn't know this was a requirement. Anyway, there's no single header for Boost.WinAPI and its headers are placed under boost/detail/winapi. Again, because it's not a public library and because it used to be part of Boost.Detail.
It's not a requirement. That should be a warning. I'll fix the checker shortly.
libs/winapi: error: directory not found; Missing [project-root]/meta directory. The [project-root]/meta directory is required for all libraries.
I thought meta/* is supposed to be created by someone else (you?) with a pull request. I'm not sure what I should do about it.
You should create it. In the past Daniel added all of them when they first introduced. But authors should be adding that file now. Also there's a python error at the end of the report:
Failed: 'b2' '--debug-configuration' '-j2' '-d0' '--check-libs-only' ERROR = 1
# /home/travis/build/boostorg/boost/status> 'b2' '--debug-configuration' '-j2' '-d0' '--check-libs-only' execution time 9.05490994453 seconds
Traceback (most recent call last):
That's just how the CI script works. It throws an exception when a command fails to avoid continuing on to other commands. When the check errors are gone that will go away. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On Wednesday, 15 June 2016 12:09:23 MSK Rene Rivera wrote:
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones those are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I plan to blindly remove that custom list of extra test targets.
core/test/swap We do have core/test/Jamfile.v2 and core/test/swap/Jamfile.v2. I think the rationale was that Boost.Swap is a sub-library of Boost.Core, and it used to have separate docs as well. Is there a way to #include core/test/swap/Jamfile.v2 into core/test/Jamfile.v2 so that Boost.Swap tests are run and displayed in the test reports?
Rene Rivera wrote:
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs https://travis-ci.org/boostorg/boost/jobs/137506433. At the moment it reports 17 libraries as having errors.
This 1564 libs/smart_ptr: warning: file found; Found extra files in [project-root]/include/boost directory. can't be fixed - it will break all code depending on these headers. This 1562 libs/smart_ptr: error: directory not found; Missing [project-root]/build directory. The [project-root]/build directory is required for libraries that have a [project-root]/src directory. is a bit of a problem as smart_ptr is header-only and requires no building, so no build/, but it has optional source files in src/. I could rename src/ to something else, I suppose.
On Wed, Jun 15, 2016 at 5:19 AM, Peter Dimov
Rene Rivera wrote:
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>. At the moment it reports 17 libraries as having errors.
This
1564 libs/smart_ptr: warning: file found; Found extra files in [project-root]/include/boost directory.
can't be fixed - it will break all code depending on these headers.
Yeah.. Just ignore the warning for now then :-) I'll see about adding a way to document expected check warnings (and hence to avoid reporting them). This
1562 libs/smart_ptr: error: directory not found; Missing [project-root]/build directory. The [project-root]/build directory is required for libraries that have a [project-root]/src directory.
is a bit of a problem as smart_ptr is header-only and requires no building, so no build/, but it has optional source files in src/. I could rename src/ to something else, I suppose.
Hmm, interesting. What are the source files for? -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
2016-06-14 17:14 GMT+02:00 Rene Rivera
On Wed, Jun 15, 2016 at 8:08 AM, Oliver Kowalke
2016-06-14 17:14 GMT+02:00 Rene Rivera
: https://travis-ci.org/boostorg/boost/jobs/135785887 reports:
# /home/travis/build/boostorg/boost/status> '../b2' '-n' '-d0'
error: Unable to find file or target named
error: '../libs/fiber/test/'
error: referred to from project at
error: '.'
... but fiber/test exists?
That's an old PR & CI test.. When fiber was not yet added to the submodules. The PR was eventually merged in after fiber was added as a submodule. And that line is by now obsolete and doesn't exists :-) As the latest equivalent build job shows < https://travis-ci.org/boostorg/boost/jobs/137712508>. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On 6/14/2016 11:14 AM, Rene Rivera wrote:
It would be great if library authors address those errors in some form ASAP. I know that for some of those reported libraries there will need to be some discussions as to how to best address the errors. Feel free to start additional threads for those as needed.
[1] < http://www.boost.org/development/tests/develop/developer/output/teeks99-09f-...
Thanks for the heads up Rene. I've pushed changes to the Jamfile.v2 for numeric/conversion. Please let me know if I've missed something. Thanks Brandon
Sorry, I'm at a loss. Wave has it's bjam file under libs/wave/test/build. What do I need to do in order to adjust its tests to the new requirements? Any help would be much appreciated. Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Rene Rivera Sent: Tuesday, June 14, 2016 10:14 AM To: boost@lists.boost.org Subject: [boost] Attention library authors.. Library requirements testing is now active.
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones those are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I plan to blindly remove that custom list of extra test targets.
2. Most structure library requirements are now checked both in the regular full testing and in the CI testing.
I've implemented a script to check for most of the library directory structure and integration requirements. The script runs as an additional CI job, currently in Travis. Here's one of those jobs < https://travis-ci.org/boostorg/boost/jobs/137506433>. At the moment it reports 17 libraries as having errors. There are also some warnings, i.e. not requirement violations but just some stuff that may be wrong. The script also runs as part of the general full testing and any problems are reported in each library's results as a single entry at the top of the tables. Here's how one of the test failures looks like < http://tinyurl.com/zbe9gal> [1] [2].
It would be great if library authors address those errors in some form ASAP. I know that for some of those reported libraries there will need to be some discussions as to how to best address the errors. Feel free to start additional threads for those as needed.
[1] < http://www.boost.org/development/tests/develop/developer/output/teeks99- 09f-win2012R2-64on64-boost-bin-v2-libs-numeric-conversion-test- __boost_check_library__-test-msvc-14-0-dbg-adrs-mdl-64-thrd-mlt.html
[2] Note there are currently some false failures in this aspect from a build system "bug". The bug is already fixed but tests will take some time to cycle to reflect the bug fix.
-- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Fri, Jun 17, 2016 at 5:58 PM, Hartmut Kaiser
Sorry, I'm at a loss. Wave has it's bjam file under libs/wave/test/build. What do I need to do in order to adjust its tests to the new requirements? Any help would be much appreciated.
For now that's fine. Some time in the past I added some requirements on test alias targets for eventual use with CI vs. non-CI testing. I don't check for those extra requirements yet so you're OK at the moment. You can see the requirements at the usual place < http://www.boost.org/development/requirements.html#Integration>. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On Fri, Jun 17, 2016 at 5:58 PM, Hartmut Kaiser
wrote: Sorry, I'm at a loss. Wave has it's bjam file under libs/wave/test/build. What do I need to do in order to adjust its tests to the new requirements? Any help would be much appreciated.
For now that's fine.
Some time in the past I added some requirements on test alias targets for eventual use with CI vs. non-CI testing. I don't check for those extra requirements yet so you're OK at the moment. You can see the requirements at the usual place < http://www.boost.org/development/requirements.html#Integration>.
Can I close the corresponding ticket, then? Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu
AMDG On 06/17/2016 05:05 PM, Rene Rivera wrote:
On Fri, Jun 17, 2016 at 5:58 PM, Hartmut Kaiser
wrote: Sorry, I'm at a loss. Wave has it's bjam file under libs/wave/test/build. What do I need to do in order to adjust its tests to the new requirements? Any help would be much appreciated.
For now that's fine.
I thought it was supposed to be in just test/ not test/build/? In Christ, Steven Watanabe
On Jun 17, 2016 7:47 PM, "Steven Watanabe"
AMDG
On 06/17/2016 05:05 PM, Rene Rivera wrote:
On Fri, Jun 17, 2016 at 5:58 PM, Hartmut Kaiser <
wrote:
Sorry, I'm at a loss. Wave has it's bjam file under
hartmut.kaiser@gmail.com> libs/wave/test/build.
What do I need to do in order to adjust its tests to the new requirements? Any help would be much appreciated.
For now that's fine.
I thought it was supposed to be in just test/ not test/build/?
Sorry, yes.. I misread Hartmut's email. Needs to have a build-able test/ dir.
On 14 June 2016 at 16:14, Rene Rivera
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones those are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I plan to blindly remove that custom list of extra test targets.
Is there any reason that you aren't supporting 'sublibs' files? They're easy enough to support, and have been in place for well over a decade.
On Sun, Jun 26, 2016 at 1:54 PM, Daniel James
As part of continued efforts to move testing to CI, to facilitate release management, and to provide more information to authors I've now implemented some testing changes:
1. Most libraries are now tested automatically if they have a usable test directory.
There are still libraries that do not follow the requirements of having all their tests in just the single test directory. You can see which ones
are here < https://github.com/boostorg/boost/blob/develop/status/Jamfile.v2#L101>. Library authors should IMMEDIATELY make changes to their test/Jamfile's to follow the library requirements for testing. Before the next release I
On 14 June 2016 at 16:14, Rene Rivera
wrote: those plan to blindly remove that custom list of extra test targets.
Is there any reason that you aren't supporting 'sublibs' files? They're easy enough to support, and have been in place for well over a decade.
It's next on my todo list.. Just forgot about that on the first implementation [1]. And i'm now busy with preparing for a trip. Should get to doing that some time with next two weeks. [1] Yes, I often forgot the special cases. Hence why I've argued in the past about not having special cases ;-) -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On 26 June 2016 at 20:03, Rene Rivera
It's next on my todo list.. Just forgot about that on the first implementation [1]. And i'm now busy with preparing for a trip. Should get to doing that some time with next two weeks.
Thanks! That's great.
On Sun, Jun 26, 2016 at 2:16 PM, Daniel James
On 26 June 2016 at 20:03, Rene Rivera
wrote: It's next on my todo list.. Just forgot about that on the first implementation [1]. And i'm now busy with preparing for a trip. Should
get
to doing that some time with next two weeks.
Thanks! That's great.
FYI.. That's now implemented on both develop and master. And for the first time in ages.. The develop and master status/Jamfile.v2 files are the same. And we are finally free of PRs against that file. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
participants (11)
-
Andrey Semashev
-
Brandon Kohn
-
Daniel James
-
Hartmut Kaiser
-
Louis Dionne
-
Oliver Kowalke
-
Peter Dimov
-
Rene Rivera
-
Soul Studios
-
Steven Watanabe
-
Tom Kent