Hi, Interesting topic. I've worked a few years ago for sometime at the vodafone testcenter in Düsseldorf, so I'm kind of familiar with testing. I'll just dump my thoughts in this mail. Its been a couple of years, so I'm not an expert in testing... And please excuse starting this mail with top-posting... ;)

Von: "Robert Ramey" An: boost@lists.boost.org Betreff: Re: [boost] How do folks test serialization code?

John Maddock wrote:

Very good questions - in ten years no one has asked this. Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging: Testing isn't standard, its the exception. At least in C++ Land.

Its nice to have all the testcases, but most companies don't cover their applications with tests. In fact, I haven't seen a single company using tests with C++ in their application in 6 years. Some of them do test, but not in codelevel. No unittests or anything like it.

The case were archives are created on one platform and loaded on another is not explicitly tested.

...

and doesn't accidently break later?

Hmmm I'm not sure how one could test that things aren't going to break later.

First one should ask what to test in this case. The user code, which uses the library, or the library it self? Of course you should handle the tests for your app loading the various versions of your format with the current/next release in testing. Simply have a couple of test archives produced by your application as a test set available for each previous version.

...

I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?

...

How does Boost.Serialization itself verify this?

It doesn't.

It also doesn't explicitly test that version x can read all archives created with versions < x.

Generally I have not had a problem with this EXCEPT in one glaring case which haunts me to this day.

So, when using serialization with boost 1.x its different then 1.y? I know creating this kind of tests is annoying, but still people should be able to upgrade their installations. But you seem to be stable enough, as I haven't heared complains that archives differ from one to another version of boost.

What the serialization library test suite DOES do is:

It has tests for all serializable data types - around 50. It runs all these tests with each archive class (4) so that's about 200 tests (+ some more for details). On my own machine, I run both the debug/release and static/dynamic versions. and msvc and gcc compilers. It does this via a tricked out bjam script file.

So the total number of tests is about 50 types * 4 archives * 2 debug/release * 2 static/dynamic * 2 compilers (gcc and msvc) or around 1600 tests. (actually it's a little more, I forget how that happened).

This takes a while. It has been referred to on this list as "carpet bombing". I took that as a complement.

I would love to see this enhanced by doing cross platform testing. Say there were 5 platforms- we could multiply the above by 25.

With running tests in the cloud that could be possible indeed, but this adds up quickly. 5 Plattforms today would maybe Win, Linux, Mac, Android & iOS. But there is much more, so testing this - while not being impossible - will keep growing into complexity, until your number of tests will not be able to run within a certain timeframe. One should also test all compilers on all available Plattforms f.e.

I would also love to see backward compatibility testing - that is test that the latest program version X could read all archives created by previous versions of the program.

I dabbled a little in this, but didnt' make much progress.

I'm actually afraid to enhance the library due to concerns that the testing requirements might be even bigger. Also I've become somewhat skeptical of the the scalability of the boost testing methodology. I'm actually working on addressing this.

That does not sound to good. Maybe its time to move on, and think about serialization II, as we have signals2 now? Would that make sense? kind regards, Jens Weller

...

Very good questions - in ten years no one has asked this. Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging: Testing isn't standard, its the exception. At least in C++ Land.

Its nice to have all the testcases, but most companies don't cover their applications with tests. In fact, I haven't seen a single company using tests with C++ in their application in 6 years. Some of them do test, but not in codelevel. No unittests or anything like it.

Pretty sad state of affairs that. I always find bugs in my code through "carpet-bomb" testing, generally in other folks code as well, just saying... John.

Very good questions - in ten years no one has asked this.

Oh :-(

...

Title almost says it all, how do you verify that your classes serialization code is truly platform independent,

This would not apply to binary archives as these are not guarenteed to be portable.

The case were archives are created on one platform and loaded on another is not explicitly tested.

...

and doesn't accidently break later?

Hmmm I'm not sure how one could test that things aren't going to break later.

Allow me to explain - I've experimented with adding serialization support to Boost.Multiprecision - amounst other things this would allow those types to be used with MPI for massively parrellel applications (but I digress). Now, binary archives are no issue, a simple loopback test is enough. But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I tend to assume that anything not tested doesn't work. Maybe I'll just create some sample archives on 32 and 64 bit platforms and try reading them. John.

...

But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I

I am interested in learning more about how this format for cpp_int is. I maintain an encoding format [1], which I would like to extend with variable-length integers.

I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc. That's what I hope anyway ;-) John.

...

I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc.

As I said before - this shouldn't be necessary.

Consider a 128-bit integer type, internally this could be: * Expressed as 8 16-bit integers (intmax_t=32 bits). * Expressed as 4 32-bit integers (intmax_t=64 bits). * Expressed as a single 128-bit (native) integer. How would you serialize from one format to a different one without breaking it down into "portably small" chunks? John.

...

How would you serialize from one format to a different one without breaking it down into "portably small" chunks?

Of course you have to define the 128 bit integer in terms of some other types. But THOSE types should already be portably serializable.

Sure, but as I said, exactly what those types are, and how many of them there are, changes from one platform/compiler to another.

another way to deal with this is to work a at a more primitive level. By marking the 128 integer type as "primitive" and defining an output for output/input stream operarate - text archives should "just work". Of course these operators will have to be implemented with portability in mind so the issue is moved to somewhere else. But then one probably needs to define these operators in any case so for text archives serialization comes for free.

You mean rely on the usual streaming << >> operators? For sure I could do that, but full string conversion is crazy expensive compared to the byte-by-byte scheme I have now.

for binary and portable binary archives, one might need to specialize the serialization operators. For binary archives there's not much if anything to do. For portable binary archives, one might have to make a minor specialization. The portable binary archive includes code for handling endianness of arbitrarily long integer types. (note that portable binary archive isn't complete in anycase in that it doesn't handle floating point types - a good GSOC project if one wants one.)

I already specialize for binary archives and just write out the bytes.

So you've got a couple of ways of addressing this.

Thanks, but now I've tested what I have (and verified that the archives are truly portable at least for 32 and 64 bit systems), I'll stick with what I have I think, Cheers, John.

...

You mean rely on the usual streaming << >> operators?

For sure I could do that, but full string conversion is crazy expensive compared to the byte-by-byte scheme I have now.

Hmm - I presume you've already had to implement these operators. So you'd be done for text based archives like text_archve and xml_archive. For these cases, I wouldn't worry about the expense as these archives are already slow enough due to all the streaming conversions they do already, I doubt anyone would notice - oh and don't forget about xml parsing. So for these archive classes:

a) performance isn't really an issue.

I hear you, but it's not quite that simple: multiprecision binary-decimal conversion is a whole minefield: * The code is terminally slow, way more than for native types. * To do it correctly for floats you need an arbitrarily large precision for the intermediate calculations in the worst case (albeit doesn't happen often). * Many libraries don't do it right anyway, so for example with mpf_t you can't round trip to and from string representations... of course that's sadly true of double on some platforms (msvc) as well :-(

b) it's helpful to be able to read floats/double, etc as human readable strings.

Can't argue with that one ;-)

c) it consumes no extra development time. - it's free!

I'm thinking the only place where making special overload would make sense would be for portable binary archive - which as far as I know doesn't currently support floating point types in any case.

Hmmm - maybe you want to generalize your method to any size float/double and include it as part of portable binary archive? That would be great!

LOL :) Not just yet, sorry! John.

Hi, On Mon, Aug 5, 2013 at 8:33 AM, Robert Ramey wrote:

...

I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?

...

How does Boost.Serialization itself verify this?

It doesn't.

It also doesn't explicitly test that version x can read all archives created with versions < x.

How about the following? 1. When new version of boost is released, add test archive file created by the version in the test/1.xx.x directory (in svn/git repo). 2. Try to test against all the previous archive when the full test run? This way, we can test the inter-version compatibility. If you want to test inter-platform, the test compares the created archive to the archive contained in the repogitory. (I am assuming that text archive is platform-independent.) I think that this will be rather easily implementable and better than nothing tested. Best regards, -- Ryo IGARASHI, Ph.D. rigarash@gmail.com