On Wed, Jun 3, 2015 at 5:46 PM, Robert Ramey
On 6/3/15 2:21 PM, Rene Rivera wrote:
No matter how bright anyone is an automated approach, as above, can't account for human nature.
well, maybe I'm just buying your argument that library authors must adhere to some reasonable conventions if they want to get their stuff tested. I think that's your original point.
Yes, that's a big point.
You pointed out as an example the multi-level directory in numeric. I guess that mislead me.
Sorry about that.. It was easier to just count and post numbers than go through the pain of enumerating individual examples. Sooooo - I'm going to support your contention that it's totally reasonable
and indeed necessary that library authors adhere to some reasonable conventions if they want thier libraries to build and test in boost. All we have to do is agree on these conventions. Here's my starting point:
Libraries should have the following directories with jamfiles in them
build test doc
Which is what we've already agreed to for more than a decade. Libraries can be nested if they adhere to the above convention.
Yep. ... add your own list here
Peter, John, and I have some more specific ones for the list. So we can agree on that. (I think).
We, as in Peter, John, you, and I, can ;-) Can't speak for others. But I would think there's some disagreement given the current variety of structures. Now I've got another issue. Why can't we just run the local testing setup
and upload the results somewhere.
It's not impossible, or even hard.
Right now I'd like users to be able to run b2 in the directories which interest them, and upload summary test results to a server which would display them.
You've mentioned this desire before :-)
This would mean that testing would be much more widespread.
It may be more widespread. But it likely will not be more varied. The current system requires that one download a python script which ...
does what?
Mostly it does a lot of error checking, fault tolerance, and handling of options (although I keep working to remove the options that are no longer used in the hope that things will get simpler).
it looks like it downloads a large number of other python scripts which then do a whole bunch of other stuff.
It downloads 3 other Python files (I could reduce that to 2 now.. just haven't gotten to it). But it also downloads the current Boost Build, process_jam_log (the C++ program), and of course Boost itself. Although it does clone the regression repo to get the related process_jam_log sources and build files.
My view is that this is and always has been misguided.
a) It's too complex
Yes, but it used to be worse.
b) too hard to understand and maintain.
Yes, but I'm trying to fix that. From multiple fronts, including investigating the option of bypassing it entirely.
c) isn't usable for the casual user
It's not meant to be as it's a serious commitment of resources to be tester for Boost. But it's easy enough to read the brief instructions and run it without knowing what it actually does.
d) requires a large amount of resources by the test
The test system resources are minuscule compared to the resources to run the tests themselves (i.e. if you just ran b2 in the status dir).
e) which he can't figure out in advance
We have experimentally arrived at resources numbers for full testing (the test scripts themselves use so little they would likely run on a modern smart phone).
f) and does a lot of unknown stuff.
I gave a presentation long ago at BoostCon #2 (IIRC), yes that far back, saying what gets done. And it does less now than it used to. But can be summarized as: 1) downloads the test system, 2) downloads Boost, 3) builds the test system, 4) builds and tests Boost, 5) processes the results, 6) uploads to a server.
Wouldn't be much easier to do something like the following:
0) download Boost (and deal with errors and proxies) a) pick a library a.2) build b2 a.3) install/setup b2 and your toolset, and possibly device or simulator or VM b) run b2
b.2) download process jam log (and deal with errors and proxies) b.3) build process jam log c) run process jam log
c.2) download X if it's not part of Boost c.3) build X d) run X which walks the tree and produces a pair of files - like
library_status does.
...Would need to produce a considerably more information laden file than what library_status does to be useful to lib authors (like what one of those Python scripts above currently does). But I understand your point. e) ftp the files to "someplace"
Like it does now.
f) the reporting system would consolidate the results and display them.
Like it does now. And of course don't forget to add a bunch of error handling (remember it's the internet, errors are everywhere), and proxy options. This would be much more flexible and easier. It would be much easier to
maintain as well.
What part would be more flexible? I don't see how it would be easier to maintain. The code would be easier to maintain? The servers easier to maintain? The report consolidation servers and code would be easier? Of course this is somewhat speculative as it's not clear to me how the
python scripts work and it's not clear how to see them without actually running the tests myself.
They pretty much work just like you described but with more automation glue :-) I've been very happy with this whole system for many years. Thank you.. But I haven't been happy with it. It works, but it has many drawbacks (none of which you mentioned). The biggest being that it suffers from low resilience. And in an ideal world I would rather see a testing system in which an individual Boost library.. 1) The authors registers to be tested on a could CI system (like Travis-CI and Appveyor) 2) Has a configuration that is standard for such testing across all of the Boost libraries. Such a configuration would: a) Automatically clone the appropriate git branch (including develop, master, PRs, whatever else you want) b) Download the latest, small, single, test script. Which would be setup from the common configuration to run for each of the cloud CI test steps. c) Download & install required software (for example it would install the version of gcc, clang, etc it's going to test with). d) Download the latest Boost Build (and build + install it) e) Download (aka git clone) the dependent Boost libraries (master by default, but could be any branch) f) Run the tests with b2. g) As part of (f) b2 "itself" would upload test results to a cloud results system live (which would process the results live and present them live) Anyway.. Test system engineering was not actually the substance of the thread. But if you want to see 1, 2.a, 2.b, 2.c, 2.d, and 2.f in action you can take a look at: https://ci.appveyor.com/project/boostorg/predef https://travis-ci.org/boostorg/predef https://github.com/boostorg/predef/blob/develop/appveyor.yml https://github.com/boostorg/predef/blob/develop/.travis.yml https://github.com/boostorg/regression/blob/develop/ci/src/script.py Note, the script.py is going to get smaller soon as there's extra code in it I though I needed as I implemented this for the past two weeks. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail