On Sun, Jan 7, 2024 at 9:23 AM Glen Fernandes via Boost < boost@lists.boost.org> wrote:

If we change what goes into the distribution, this is an option. As far as I was told, at our current distribution size, this would require LFS which GitHub would charge us for.

What would the file sizes look like if the release came out as two separate archives: One with complete HTML documentation, and the other with the source code? Thanks

On Sun, Jan 7, 2024 at 12:32 PM Peter Dimov wrote:

Glen Fernandes wrote:

...

If we change what goes into the distribution, this is an option. As far as I was told, at our current distribution size, this would require LFS which GitHub would charge us for.

https://docs.github.com/en/repositories/releasing-projects-on-github/about-r...

says

"Each file included in a release must be under 2 GiB. There is no limit on the total size of a release, nor bandwidth usage."

The currently hosted archives are comparable in size with the official releases.

The official boost_1_84_0.7z is 106 MB, and the corresponding CMake archive is 90.1 MB.

In other words, as long as the GitHub release can be made from our existing repository contents, we should be fine? i.e. We cannot put our current official built releases into a GitHub repository because any file over 100 MB would be rejected: https://docs.github.com/en/repositories/working-with-files/managing-large-fi... "GitHub blocks files larger than 100 MiB. To track files beyond this limit, you must use Git Large File Storage (Git LFS)." Glen

On Sun, Jan 7, 2024 at 1:15 PM Peter Dimov wrote:

...

On Sun, Jan 7, 2024 at 12:32 PM Peter Dimov wrote:

Glen Fernandes wrote: > If we change what goes into the distribution, this is an option. As far as I was > told, at our current distribution size, this would require LFS which GitHub > would charge us for.

https://docs.github.com/en/repositories/releasing-projects-on- github/about-releases

says

"Each file included in a release must be under 2 GiB. There is no

Glen Fernandes wrote: limit on

...

the total size of a release, nor bandwidth usage."

The currently hosted archives are comparable in size with the

official

...

releases.

The official boost_1_84_0.7z is 106 MB, and the corresponding CMake archive is 90.1 MB.

In other words, as long as the GitHub release can be made from our existing repository contents, we should be fine?

i.e. We cannot put our current official built releases into a GitHub repository because any file over 100 MB would be rejected:

https://docs.github.com/en/repositories/working-with-files/managing-large-

...

files/about-large-files-on-github

"GitHub blocks files larger than 100 MiB. To track files beyond this limit, you must use Git Large File Storage (Git LFS)."

https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84....

is 149 MB.

The above probably refers to putting large files in a repository, not to release artifacts.

Thanks; then leveraging GitHub releases seems like the best solution to me. Maybe the only (free) solution we have. Glen

On 1/8/24 16:09, Sam Darwin via Boost wrote:

...

Thanks; then leveraging GitHub releases seems like the best solution to me. Maybe the only (free) solution we have.

Glen, While GitHub Releases seems like the best plan, here are a few other details. JFrog is hosting: 1. development snapshots 2. beta and release-candidates 3. releases

The GitHub Releases paradigm is well designed for 3, and possibly 2. However it's not designed for "development snapshots" and would really be forcing that functionality.

What's the purpose of development snapshots? Can we drop them in favor of git checkout?

On 1/8/24 18:53, Sam Darwin wrote:

...

Cloudflare

2.8 Limitation on Serving Non-HTML Content The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages 

...

What's the purpose of development snapshots?

- The existing scripts that publish official releases do that by using the development snapshots. Renaming the latest snapshot, which was built using CircleCI. - Constantly validating the full build process through CircleCI - 'develop' and 'master' docs are sourced from the snapshots. Notice the 'develop' in this URL. https://www.boost.org/doc/libs/develop/tools/boostdep/doc/html/ https://www.boost.org/doc/libs/develop/tools/boostdep/doc/html/   - a 'git checkout' would still need to compile the html files. After that has been done, it's convenient to zip the results, and upload them to the web server.

Boost's internal processes (e.g. that the website uses snapshots to publish docs from develop and master) can be changed, if needed. That is, if snapshots are problematic to serve, and the sole their user is Boost itself, we should be able to do without them. For example, update the website from CI itself. I was more interested in whether the actual users need them.

Sam Darwin wrote:

...

Cloudflare

2.8 Limitation on Serving Non-HTML Content The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages

https://blog.cloudflare.com/updated-tos/ "Goodbye, section 2.8 and hello to Cloudflare’s new terms of service 16/05/2023" "Cloudflare’s network became larger and more robust and its portfolio broadened to include services like Stream, Images, and R2. These services are explicitly designed to allow customers to serve non-HTML content like video, images, and other large files hosted directly by Cloudflare. ... we made it clear that customers can serve video and other large files using the CDN so long as that content is hosted by a Cloudflare service like Stream, Images, or R2." Phil.

On 1/8/24 18:55, Vinnie Falco wrote:

On Mon, Jan 8, 2024 at 7:44 AM Andrey Semashev via Boost mailto:boost@lists.boost.org> wrote:

What's the purpose of development snapshots? Can we drop them in favor of git checkout?

The development snapshot is effectively a release. It has all the headers collated into the proper root include directory. It has all the sources in the right place for building each lib. It has the rendered HTML. The snapshot allows anyone to efficiently test whether or not an issue is resolved in the tip of a branch for example.

He could also test a git checkout.

"git checkout" doesn't provide anything remotely resembling a proper release. To function as a drop-in replacement for an installed official release of Boost would require a lot of post-processing after checking out the superproject using git.

Is this functionality actually in high demand by users? I would imagine, the majority of users either consume Boost releases (or betas or RCs), or use Boost from git.

Andrey Semashev wrote:

What's the purpose of development snapshots? Can we drop them in favor of git checkout?

Website documentation snapshots for master and develop are created from them.

I wrote:

the deactivation of the JFrog account in December 31 was not expected, and it was reinstated.

Our JFrog downloads are back. JFrog had renewed our account for an extra week instead of an extra year. Glen

On Monday, January 8, 2024, Ion Gaztañaga wrote:

Hi Glen,

Maybe this issue has been already discussed in the Boost Foundation. According to the minutes, some cost reduction efforts were discussed in the past (Mailing list, Server cost...). This mainly looks like a cost issue to me (if I understand your description correctly, it's not an storage problem but a network bandwidth problem)

Binary download is the primary Boost distribution method, maybe we should try to get some funding or sponsorship through the Boost Foundation. Does it make sense?

Hi Ion, Yes, we have been negotiating for an affordable rate, the most competitive of which we will have a formal quote for on Wednesday (01/10). A free solution or free fallback is still preferable even if it only covers releases, RCs, and betas (and not development snapshots etc.). Yes, binary download here being our tarballs hosted on JFrog currently (not the static libraries built by Tom). Glen

by a Cloudflare service like Stream, Images, or R2.

Interesting. A CDN would still be restricted. But if we migrated the files to their R2 service then it would be permitted. In the longer term that could be worth exploring.

> Perhaps it's time to "get serious" about Boost "Modularization". 
> Basically this would mean that users download just the libraries (and 
> their dependencies) they actually intend to use.  Of course this would 
> be a big project.  But we've been working hard to try to move in this 
> direction.  I would envision:
>
> a) user interested in boost download and locally test Boost "core"
> b) for each library that a users is immediately interested in:
>  downloads, builds and tests the library (and it's dependencies)
> c) as time moves on, users could update, replace, or delete their set 
> of libraries.
>
> This would in practice eliminate the concept of Boost version 1.84 
> etc... and replace with Boost Serialization library version 1, ...
> Boost would migrate from being a single/monolithic library to a group 
> of libraries with some explicit dependencies (on other boost librarys, 
> standard library or ?). 
While I agree that this modularization would be great and helpful 
especially for package managers (and their maintainers) I don't think 
distributing Boost libs as a loose collection of libraries is good for 
end-users/developers:
- Boost libs do and should depend on other Boost libs especially for 
compatibility, bug fixes, improvements etc over the stdlib/compiler
- Dependencies of Boost libraries are not obvious, especially the 
transitive ones making it hard to keep a working configuration
- Dependencies might change with or without notice
- It could make users believe they can mix and match Boost libraries of 
different versions while we do (and possibly can) only test a single 
configuration, i.e. the current state of master/develop/tagX of ALL 
libraries at that state
- Although that "monolith" thing is a common complaint it also has an 
advantage: Once you have Boost already downloaded/set up you have 
something close to an "extended standard library" and are encouraged to 
look if what you need is already in one of the Boost libs you already 
have available instead of rolling your own. Having to go through the 
trouble of getting yet another set of dependent libraries is likely 
off-putting.

Especially for the "with some explicit dependencies (on other boost 
librarys" part:
Having the experience trying to just have a CMakeList (for CI testing) 
listing all direct and transitive dependencies of a Boost library "at 
the bottom" has shown that this is quite fragile and has led to trouble 
before. Pushing that to users isn't an idea I'm very fond off. And as a 
Boost maintainer I don't really want to care if some other Boost lib did 
or did not add or remove a dependency on another Boost lib.
So having that list of "explicit dependencies" will be hard to keep up 
to date and valid at all times. Currently we can get away by saying 
"Boost is the dependency of Boost"
Yes the dependency tool (e.g. used in CI) does a good job already and we 
could leverage that to create that list. It will still be changing and 
would need to be much more reliable, which means maintainers need to do 
more work getting their (direct) dependencies always(!) right and in the 
format that tool understands.

However I do like the approach we currently have in CMake (and likely 
soon in B2) of not requiring a single "include/boost" folder but having 
the build system figure out (transitive) dependencies and required paths.

Alex

On 1/10/24 6:38 AM, Vinnie Falco via Boost wrote:

On Tue, Jan 9, 2024 at 11:49 AM Robert Ramey via Boost < boost@lists.boost.org> wrote:

...

Perhaps it's time to "get serious" about Boost "Modularization".

This statement ignores all the work that has been done and continues to be done in terms of making Boost modular.

Just the opposite. It acknowledges the work done and implicitly laments that it has failed to arrive at its logical conclusion. What is the purpose of investing the effort into "Boost Modularization" if it's not this?

...

Basically this would mean that users download just the libraries (and their dependencies) they actually intend to use. Of course this would be a big project. But we've been working hard to try to move in this direction.

Yes, we have "been serious." (And note when I say "we" I exclude myself, as I have just been busy cranking away at producing individual libraries and supporting Boost Libraries infrastructure).

...

This would in practice eliminate the concept of Boost version 1.84 etc... and replace with Boost Serialization library version 1, ...

Grouping all the libraries together into a single release version (e.g. 1.85.0) which is all tested against each other as a unit, is the only sane development model. Otherwise we run into the combinatorial explosion of questions like "what version works with what."

Actually, I believe this statement is exactly wrong. a) Treating Boost "as a unit" and testing on this basis results in an amount of work which increases with the square of the number of libraries. b) Modifying libraries to pass tests in other libraries make one library dependent on another which might not be obvious. Libraries should be tested individually as a unit to prove that the the implementation of the library matches faithfully implements it's expored interface. c) If this is done, it is guaranteed that errors cannot be introduced when libraries are composed. d) Libraries interfaces including type requirements should be formally documented and tests should certify that the implementation is consistent with the interface. e) If the above is done - the amount of testing will increase only linearly with the number of libraries. f) Adding one more library should not provoke any new errors. e) Introducing an error into a library should be detected during unit testing and if inspite of this it's undetected, it will only affect other libraries which import the header from the erroneous library. g) Library authors should strive to detect at compile time (preferibly) or runtime which interface requirements are not met. Boost testing has special facilities "test compile fail" specifically to facilitate that. h) The need to "test all the libraries together" is a red flag/code smell. In any case, users should be able to download any number and/or combination of libraries (along with their dependencies) and use just that. This will avoid making uses applications more complicated than they already are. And furthermore,

"eliminating the concept of Boost version ${X}" pushes more testing and documentation work (to explain what versions work with what) onto each individual author instead of centralizing that effort into the release process. I don't like this at all.

Right - Wrong ! If we need to specify that information - we've already made a mistake. No one can keep all those combinations in their head. If some library/app depends on some other library subject to some dependency of compiler level, etc. and that requirement is unfullfilled, It should result in a compile time error. Our config library - a masterpiece by John Maddock - is designed to address this very problem and does so very well.

Boost would migrate from being a single/monolithic library to a group of

...

libraries with some explicit dependencies (on other boost librarys, standard library or ?).

It might. I can't predict the future. Nothing in this proposition precludes a "complete release". But it should be an option - not a requirement.

...

Boost is already a "group of libraries with some explicit dependencies." It just so happens that they are bundled together into one archive. Whatever it is that you are proposing would be in addition to and not in lieu of what we have.

Sure. But if we finished the "boost modularization", the "global" release would be nothing more than the union of the individual ones and guarenteed to be correct.

...

The fact that we can't do so now is a symptom that our development practices need work.

I think this is overlooking the fact that the Boost release process *works well* right now. Three releases every year like clockwork and they are pretty high quality in terms of having minimal inter-library defects.

I don't dispute this. But - it doesn't scale and can never scale. That's what started this discussion in the first place. I'm aware that significant effort has been invested into the "boost modularization" effort. I have a couple of questions about this effort. a) what is the point of this "modularization" effort if not this? b) what this the desired/expected benefit of this effort if not this? c) when will we know when it's done? d) how will we know whether or not it's been successful?

...

Robert Ramey

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 1/11/24 9:26 PM, Alan de Freitas via Boost wrote:

...

a) Treating Boost "as a unit" and testing on this basis results in an amount of work which increases with the square of the number of libraries.

Sorry. Why exactly the square of the number of libraries?

Suppose you've got one library with 10 cases you want to test and each test takes 1 second to run. Now suppose you've got 2 libraries each with 10 cases. Suppose you're concerned about one library provoking a failure in the other. Then for each test in the first library, there might be 10 conditions in the second library which you would want to test against. etc... Actually, a better analysis might conclude that the number of possible cross failure modes might increase with the following number: n + n * (n-1) / 2 + n * (n-1) * (n-2) / 3, + n * (n-1) * (n-2) * (n-3) / 4 ... Of course it's a crude measure (and argument). But it illustrates that if you're trying to test cross impacts of libraries, the number of possible failure modes increases disproportionately to the number libraries to be tested. Actually when we think we're "cross testing" we're really not because we aren't really writing tests to consider these kinds of failures. So the whole idea of thinking that we're actually testing anything when we test all at once is very misleading. A related situation occurs when making a scientific experiment. Typically such an experiment has a control case and the test case which varies from the control case in only one variable. So if the two cases result in different results, we know that that one variable is the source of the difference. Trying to test "all at once" is exactly the opposite of scientific method. The whole idea of unit testing is an attempt to make our testing more useful and scientific. In the "old days", we would write the whole program from start to finish before we did any testing. This is comparable to the "cross testing" argument from earlier in this post. This wasn't called "testing" it was called "debugging". It proved to be a very inefficient and time consuming operation. In reference to the above, consider how much more time it takes to "debug" the whole program as opposed to testing each function/type individually. As yet another aside, I worked for years as a freelance developer/consultant. I only got called when things were stuck and they needed some to take the blame and they had no other choice. Part of this was likely due to my annoying and pedantic personality. I have never had a customer who ever wrote unit test. When I asked why, the answer was always "we haven't got time". Historically, the idea of unit testing only really became a "thing" around the year 2000. Imagine - 30-40 years of software development with the build and crash method. Another historical note that I believe that I'm repeating correctly. When the first stored program computer was fired up they tried a program like factoring a number or something. They (including John von Neuman), were astonished that it didn't work the first time!!! Given the mindset of my collegues this doesn't amaze me. Another interesting note from the past was that up until ~1960 programmers were almost all female. It didn't take long (~10 years) before most of them were men. I have no idea why this is/was. Make of this whatever you want. I'm sure someone will have a theory. Robert Ramey

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 1/12/24 05:16, Robert Ramey via Boost wrote:

a) Treating Boost "as a unit" and testing on this basis results in an amount of work which increases with the square of the number of libraries. b) Modifying libraries to pass tests in other libraries make one library dependent on another which might not be obvious.  Libraries should be tested individually as a unit to prove that the the implementation of the library matches faithfully implements it's expored interface. c) If this is done, it is guaranteed that errors cannot be introduced when libraries are composed.

No, it doesn't. Even disregarding that you can't reasonably test everything, you're forgetting that libraries change (yes, including the public interface) and often affect each other through their usage, sometimes in non-obvious ways. Suppose, a library A provides a unique_ptr implementation (such as the one in Boost.Move) that supports a custom deleter. That library may test that unique_ptr does use the deleter as intended. But that doesn't guarantee that this will still work in another library B that uses the unique_ptr with its custom deleter - for example, because that deleter is defined in B's namespace and since it is specified in unique_ptr template parameters, it now affects ADL. Integration testing exists for a reason. If you're not doing integration testing, you're getting a bunch of disparate components that don't compose well or at all.

Robert Ramey wrote:

This would in practice eliminate the concept of Boost version 1.84 etc... and replace with Boost Serialization library version 1, ...

For most intents and purposes, Boost releases _are_ Boost. If we eliminate the concept of Boost release 1.84, what remains already exists and is called Github.

Boost would migrate from being a single/monolithic library to a group of libraries with some explicit dependencies (on other boost librarys, standard library or ?).

Boost would migrate into nothing.

On 1/13/24 8:35 AM, Janek Kozicki via Boost wrote:

Robert Ramey via Boost said: (by the date of Tue, 9 Jan 2024 11:48:44 -0800)

...

This would in practice eliminate the concept of Boost version 1.84 etc... and replace with Boost Serialization library version 1, ... Boost would migrate from being a single/monolithic library to a group of libraries with some explicit dependencies (on other boost librarys, standard library or ?).

I have a following code snippet in my serialization header:

// To correctly recognize serialization of Inf and NaN numbers, // there are different includes for different boost versions #if BOOST_VERSION>=104700 #include #else #include #endif

How do you plan to not break it with separate version numbers for each library?

I would expect each library to be versioned individually. So the above would look something like:

// To correctly recognize serialization of Inf and NaN numbers, // there are different includes for different boost versions #include #if BOOST_MATH_VERSION >= 12 (or ?) #include #else #include #endif

As an (off topic) aside, that the the serialization library has an archive version number embedded in every archive. It's incremented everytime the archive format is amended. In this way the most recent versions of the serialization library can read archives created in all previous versions of the library. I think this number is up to 20 now. Robert Ramey

best regards Janek

-- Janek Kozicki, PhD. DSc. Arch. Assoc. Prof. Gdansk University of Technology (Gdansk Tech) Faculty of Applied Physics and Mathematics Institute of Physics and Applied Computer Science Division of Theoretical Physics and Quantum Information -- http://yade-dem.org/ http://pg.edu.pl/p/jan-kozicki-19725 http://mostwiedzy.pl/en/jan-kozicki,19725-1

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost