Vinnie Falco via Boost writes:

I guess what I'm trying to say here is that we care deeply about long turnaround times on CI ("they suck) have been working on the problem.

This email may give an impression that the CI suckage is an inherent problem that one just has to live with so I want to provide a different perspective. As part of the build2 project we have implemented our own CI infrastructure from the grounds up: from managing our own hardware, to VM-based CI job handling, to custom web interface for result viewing. This was based on concluding that all the existing off- the-shelf solutions are primarily focused on "web development" and will result in the subpar experience for large C/C++ projects (Boost, Qt, etc). Some key differences of our approach compared to the commonly used setups (e.g., GitHub Actions): 1. "Push CI" explicitly triggered during development as opposed to "pull CI" triggered on commit/PR/etc. This allows specifying the target configurations to build in, alternative project configurations, etc.[1] 2. Standard "CI script"[2] that is used by all the C/C++ projects instead of each projects doing their own thing. This "CI script" is written in C++ and has evolved to cover a fairly elaborate set of scenarios, including: 1. Testing of preparation of source distributions (to make sure there are no missing files, etc). 2. Testing of libraries as installed (to make sure no public headers were omitted during installation, etc). 3. Ability to test projects with build-time dependencies (e.g., source code generators, build system modules, etc) in cross- compiled environments. 3. Interactive CI support. The push CI can request an interactive session where the above "CI script" is stopped at one of the pre-defined breakpoints allowing the user to login into the machine via VNC or SSH and investigate the problem. 4. Large number of build configurations covering all the major platforms/compilers/versions[3] with a well-defined obsolescence policy. To give a sense of how this works in practice, here is my typical workflow: 1. Hack on some feature in a branch, build and test locally. 2. When need to test on other platforms, push the branch to remote and submit the CI request. Normally it's just: $ bdep ci The result is a URL with the interface like this: https://ci.stage.build2.org/@04e5ffb7-cf96-42f9-bc91-94d3ffe48de1 One nice touch is that it detects compiler warnings, as can be seen in this result: https://ci.stage.build2.org/@23da340d-cb4d-4105-8474-bb870e1a1380 (Drop the UUID from the above URL to see some other CI jobs.) 3. Say everything works fine except on Windows with an old MSVC version. If the fix is clear, I push it and re-CI but just on this specific configuration (this is both faster for me and saves resources for others): $ bdep ci --build-config windows_10-msvc_14.3 If the fix is not clear and I need to play with it or run the debugger, I can request an interactive CI session: $ bdep ci --interactive windows_10-msvc_14.3/error That `/error` part is a breakpoint, meaning that the CI job will execute the standard "CI script" until the first error (compile error, test failure, etc) and pause. 4. Once all is good I clean up my feature branch (squash commits, etc), rebase on master, CI one more time for good measure, and merge to master. When I push my changes to master, there can be a hook that submits a "verification" CI job. [1] https://build2.org/bdep/doc/bdep-ci.xhtml [2] https://build2.org/bbot/doc/build2-build-bot-manual.xhtml#arch-worker [3] https://ci.cppget.org/?build-configs