Boost.Uuid and header-only support

older
Preferred way to simulate system...

James E. King, III

3 Nov 2017 3 Nov '17

11:58 p.m.

Hi folks, I have an open defect related to a random seed generator that's currently in boost::uuids::detail called seed_rng. The class does a fair bit more than it really needs to, and does not support Universal Windows Platform (UWP) store targets. It is otherwise quite similar to boost::random::random_device however it is a header-only implementation (in fact seed_rng has a comment about putting it into Boost.Random). I was planning on removing it and leaving randomness to Boost.Random because that's where the code really belongs, however this would introduce a new library dependency on Boost.Random for any code using boost::uuids::random_generator. So I'm curious on how important folks believe it is to maintain the header-only status of Boost.Uuid. Thanks, Jim

Show replies by date

Peter Dimov

4 Nov 4 Nov

12:15 a.m.

James E. King, III wrote:

...

Hi folks,

I have an open defect related to a random seed generator that's currently in boost::uuids::detail called seed_rng. The class does a fair bit more than it really needs to, and does not support Universal Windows Platform (UWP) store targets. It is otherwise quite similar to boost::random::random_device however it is a header-only implementation (in fact seed_rng has a comment about putting it into Boost.Random). I was planning on removing it and leaving randomness to Boost.Random because that's where the code really belongs, however this would introduce a new library dependency on Boost.Random for any code using boost::uuids::random_generator. So I'm curious on how important folks believe it is to maintain the header-only status of Boost.Uuid.

You had plans of contributing a header-only random_device to Boost.Random, I think?

Andrey Semashev

11 a.m.

On 11/04/17 03:15, Peter Dimov via Boost wrote:

...

James E. King, III wrote:

...
Hi folks,

I have an open defect related to a random seed generator that's currently in boost::uuids::detail called seed_rng. The class does a fair bit more than it really needs to, and does not support Universal Windows Platform (UWP) store targets. It is otherwise quite similar to boost::random::random_device however it is a header-only implementation (in fact seed_rng has a comment about putting it into Boost.Random). I was planning on removing it and leaving randomness to Boost.Random because that's where the code really belongs, however this would introduce a new library dependency on Boost.Random for any code using boost::uuids::random_generator. So I'm curious on how important folks believe it is to maintain the header-only status of Boost.Uuid.

You had plans of contributing a header-only random_device to Boost.Random, I think?

The proposal to make Boost.Random header-only was rejected by Steven. https://github.com/boostorg/random/pull/29

Peter Dimov

3:40 p.m.

Andrey Semashev wrote:

...

The proposal to make Boost.Random header-only was rejected by Steven.

https://github.com/boostorg/random/pull/29

This sounds like essential context that might have been worth mentioning. :-) Steven is right that we're still dependent on Boost.System, this needs to be solved somehow. To get back to Uuid. The whole seed_rng file needs to be retired. It's only an implementation detail of a primitive operation that returns N random bytes. At the time seed_rng was written, there was no good way of obtaining high quality randomness from the OS, but this is not the case today. What the Uuid library needs, when generating a random UUID, is just an array of random bytes; there's no need to create a random number generator, then seed it. None at all. This is historical baggage. And in my opinion, all the conceptification and templatization in James's PR is completely unnecessary and just adds complexity. What we need is this: namespace boost { int get_random_bytes( void * buffer, size_t size ); // returns errno } This happens to match the Linux function with the same name but this is coincidental. The implementation of this function needs to - read from /dev/urandom on POSIX - use RtlGenRandom on Windows desktop - use Bcrypt on Windows non-desktop We have to decide where to put it though. Utility springs to mind as a first candidate. Switching topics back to Boost.Random, I consider the use of the token to select a cryptographic provider a bad practice. Even the POSIX path should ignore the token. get_random_bytes is the way to go. But that's a separate story.

Andrey Semashev

4:57 p.m.

On 11/04/17 18:40, Peter Dimov via Boost wrote:

...

Andrey Semashev wrote:

...
The proposal to make Boost.Random header-only was rejected by Steven.

https://github.com/boostorg/random/pull/29

This sounds like essential context that might have been worth mentioning. :-)

Steven is right that we're still dependent on Boost.System, this needs to be solved somehow.

I think, Steven was more opposed to the whole idea of header-only code, regardless of implementation.

...

To get back to Uuid. The whole seed_rng file needs to be retired. It's only an implementation detail of a primitive operation that returns N random bytes. At the time seed_rng was written, there was no good way of obtaining high quality randomness from the OS, but this is not the case today.

What the Uuid library needs, when generating a random UUID, is just an array of random bytes; there's no need to create a random number generator, then seed it. None at all. This is historical baggage.

Well, not exactly. I agree that some code in seed_rng.hpp is unneccessary and outdated. But certainly not all of it. Boost.UUID needs a good PRNG, preferably a fast one. An RNG that blocks if there's not enough entropy is not suitable. If OS provides such PRNG then great, we can use it. I'm not sure every platform does, though. In particular, I'm not sure Windows APIs guarantee the non-blocking behavior. Another point to consider is whether we want to deplete system entropy pool when generating lots of UUIDs. The primary property we need from random numbers is uniqueness; we don't require cryprographical randomness for every byte of an UUID or maybe even every generated UUID.

...

And in my opinion, all the conceptification and templatization in James's PR is completely unnecessary and just adds complexity. What we need is this:

namespace boost { int get_random_bytes( void * buffer, size_t size ); // returns errno }

This happens to match the Linux function with the same name but this is coincidental.

The implementation of this function needs to

- read from /dev/urandom on POSIX

There are also Linux-specific getrandom and BSD-specific getentropy.

...

- use RtlGenRandom on Windows desktop

IIRC, it's only available since Vista or 7? I forget.

...

- use Bcrypt on Windows non-desktop

The devil is in the details, as usually. The function is not the proper interface for such a tool because, e.g. in case of /dev/urandom-based implementation, it will have to open/close a file on every call. This is an additional source of failure and an attack vector. The standard interface for system RNG is std::random_device, which you can create and use multiple times. Boost.UUID also rightfully implements random_generator as an object to accommodate this usage pattern.

...

Switching topics back to Boost.Random, I consider the use of the token to select a cryptographic provider a bad practice. Even the POSIX path should ignore the token. get_random_bytes is the way to go. But that's a separate story.

I'm not sure I understand what you mean here. I agree that APIs like getrandom and getentropy are superior to reading from /dev/urandom, but those are not available universally. I'm not sure how that relates to Windows though.

Edward Diener

5:16 p.m.

On 11/4/2017 12:57 PM, Andrey Semashev via Boost wrote:

...

On 11/04/17 18:40, Peter Dimov via Boost wrote:

...
Andrey Semashev wrote:

...
The proposal to make Boost.Random header-only was rejected by Steven.

https://github.com/boostorg/random/pull/29

This sounds like essential context that might have been worth mentioning. :-)

Steven is right that we're still dependent on Boost.System, this needs to be solved somehow.

I think, Steven was more opposed to the whole idea of header-only code, regardless of implementation.

Steven can chime in if he wants but my interpretation was that Steven was rightly concerned with the idea that making Boost random header-only only works if the end-user is forced to use Boost system as header-only, and we do not want to tell the end-user what he must do. snipped...

James E. King, III

5:47 p.m.

On Sat, Nov 4, 2017 at 1:16 PM, Edward Diener via Boost < boost@lists.boost.org> wrote:

...

On 11/4/2017 12:57 PM, Andrey Semashev via Boost wrote:

...
On 11/04/17 18:40, Peter Dimov via Boost wrote:

...
Andrey Semashev wrote:

The proposal to make Boost.Random header-only was rejected by Steven.

...
https://github.com/boostorg/random/pull/29

This sounds like essential context that might have been worth mentioning. :-)

Steven is right that we're still dependent on Boost.System, this needs to be solved somehow.

I think, Steven was more opposed to the whole idea of header-only code, regardless of implementation.

Steven can chime in if he wants but my interpretation was that Steven was rightly concerned with the idea that making Boost random header-only only works if the end-user is forced to use Boost system as header-only, and we do not want to tell the end-user what he must do.

Making Boost.Random header-only could still be considered an improvement, whether or not what it depends on is header-only. Perhaps defining BOOST_ERROR_CODE_HEADER_ONLY would allow someone to use both libraries in a header-only environment? I'm not sure. Given that random_device is the only implementation that uses system_error, one could make the errors in random_device more abstract with its own exception type, i.e. boost::random::entropy_error : public std::runtime_error, and sever the link between the libraries, and then a header-only implementation of random would be immediately useful... - Jim

Peter Dimov

5:53 p.m.

James E. King, III wrote:

...

Making Boost.Random header-only could still be considered an improvement, whether or not what it depends on is header-only.

It might, but Uuid will still regress from header-only to requiring a library, so this doesn't help.

James E. King, III

5 Nov 5 Nov

12:31 a.m.

On Sat, Nov 4, 2017 at 1:53 PM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

Making Boost.Random header-only could still be considered an improvement,

...
whether or not what it depends on is header-only.

It might, but Uuid will still regress from header-only to requiring a library, so this doesn't help.

I was wondering, does it even make sense to have the default RNG of uuids::random_generator set to a PseudoRandomNumberGenerator for boost::uuid? It is very expensive to seed and then use a mersenne twister when compared to just acquiring 16 bytes of entropy. I'd like the default uuids::random_generator to use a random_device implementation (I have successfully moved the implementation I have into uuid detail locally), however if I change the default type in uuids::random_generator that would be a breaking change for anyone using the constructors that take a reference or a pointer... it would be much more efficient to just get 16 bytes of entropy instead.

Peter Dimov

12:53 a.m.

James E. King, III wrote:

...

I was wondering, does it even make sense to have the default RNG of uuids::random_generator set to a PseudoRandomNumberGenerator for boost::uuid?

No, in my opinion it doesn't. basic_random_generator has to be retained for compatibility, but random_generator should just obtain random bytes directly. You're right that this is a breaking change though - a justified one, in my opinion.

James E. King, III

1:21 a.m.

On Sat, Nov 4, 2017 at 8:53 PM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

...
I was wondering, does it even make sense to have the default RNG of uuids::random_generator set to a PseudoRandomNumberGenerator for boost::uuid?

No, in my opinion it doesn't. basic_random_generator has to be retained for compatibility, but random_generator should just obtain random bytes directly. You're right that this is a breaking change though - a justified one, in my opinion.

Looks like November 1 was the deadline for making major changes for 1.66.0, which I assume would include breaking changes...

Peter Dimov

1:37 a.m.

James E. King, III wrote:

...

On Sat, Nov 4, 2017 at 8:53 PM, Peter Dimov via Boost
...
James E. King, III wrote:

...
I was wondering, does it even make sense to have the default RNG of uuids::random_generator set to a PseudoRandomNumberGenerator for boost::uuid?

No, in my opinion it doesn't. basic_random_generator has to be retained for compatibility, but random_generator should just obtain random bytes directly. You're right that this is a breaking change though - a justified one, in my opinion.

Looks like November 1 was the deadline for making major changes for 1.66.0, which I assume would include breaking changes...

That's true, in principle. The intent behind the rule is however that this refers to changes that break other Boost libraries, not client code outside of Boost. Client code typically sees the breaking change after the release, or at best after the beta, not before. So breaking client code does not in general impede the release process, whereas breaking other Boost libraries (or otherwise introducing regressions into the testing and release procedures) most certainly can and does. But a permission from a release manager cannot hurt if you decide to pursue this.

James E. King, III

2:53 a.m.

On Sat, Nov 4, 2017 at 9:37 PM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

...
On Sat, Nov 4, 2017 at 8:53 PM, Peter Dimov via Boost < boost@lists.boost.org wrote:

...
James E. King, III wrote:

...
I was wondering, does it even make sense to have the default RNG of >> uuids::random_generator set to a PseudoRandomNumberGenerator for >> boost::uuid?

No, in my opinion it doesn't. basic_random_generator has to be retained for compatibility, but random_generator should just obtain random bytes > directly. You're right that this is a breaking change though - a > justified one, in my opinion.

Looks like November 1 was the deadline for making major changes for 1.66.0, which I assume would include breaking changes...

That's true, in principle.

The intent behind the rule is however that this refers to changes that break other Boost libraries, not client code outside of Boost. Client code typically sees the breaking change after the release, or at best after the beta, not before. So breaking client code does not in general impede the release process, whereas breaking other Boost libraries (or otherwise introducing regressions into the testing and release procedures) most certainly can and does.

But a permission from a release manager cannot hurt if you decide to pursue this.

I did a short test where I use the old mt19937 based generator in a loop to make 1 million uuids and then I compared that to a new random_device implementation (both under basic_random_generator using the default constructor), and the results surprised me. On a Windows 10 laptop with a Xeon processor, using Visual Studio 2017 and building release: Benchmark Times reusing a generator (1M loops): old implementation: 0.021822s wall, 0.031250s user + 0.000000s system = 0.031250s CPU (143.2%) new implementation: 0.373160s wall, 0.375000s user + 0.000000s system = 0.375000s CPU (100.5%) Benchmark Times using a new generator for each uuid (10K loops): old implementation: 1.168479s wall, 1.171875s user + 0.000000s system = 1.171875s CPU (100.3%) new implementation: 0.010272s wall, 0.015625s user + 0.000000s system = 0.015625s CPU (152.1%) Asking the OS for 16 bytes of entropy at a time (Wincrypt, 1M loops): 0.244231s wall, 0.234375s user + 0.000000s system = 0.234375s CPU (96.0%) So the initialization of the mersenne twister is expensive, so if you do it once and then reuse it, it is much more efficient than going to Wincrypt. If you make a new generator for each uuid, the opposite is true. I expect folks will do the former, but I've seen the latter in some projects. It looks like it is very expensive to get entropy through Wincrypt (and BCrypt), whereas using it only to seed a PRNG and then reusing the PRNG it is much more efficient through the existing code paths. Therefore keeping the default type at mt19937 and using this newer header-only random_device will maintain performance and compatibility, but also fix the problems in seed_rng like how it would ignore errors, and will add UWP support. Even asking Wincrypt for 16 bytes of entropy at a time is not efficient, so rewriting the implementation to be less generic won't improve performance. So in short, I'll plan to keep the same implementation of random_device but put in a new seed mechanism for the PseudoRandomNumberGenerator. I also fixed up the template specialization that was in seed_rng so that any UniformRandomNumberGenerator can be used. Just not sure this will go into 1.66.0 at this point... might be too big of a structural change internally. - Jim

Peter Dimov

2:59 a.m.

James E. King, III wrote:

...

Benchmark Times reusing a generator (1M loops): old implementation: 0.021822s wall, 0.031250s user + 0.000000s system = 0.031250s CPU (143.2%) new implementation: 0.373160s wall, 0.375000s user + 0.000000s system = 0.375000s CPU (100.5%)

Benchmark Times using a new generator for each uuid (10K loops): old implementation: 1.168479s wall, 1.171875s user + 0.000000s system = 1.171875s CPU (100.3%) new implementation: 0.010272s wall, 0.015625s user + 0.000000s system = 0.015625s CPU (152.1%)

These results look odd to me. What code exactly is being tested?

James E. King, III

4:37 a.m.

On Sat, Nov 4, 2017 at 10:59 PM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...

James E. King, III wrote:

Benchmark Times reusing a generator (1M loops):

...
old implementation: 0.021822s wall, 0.031250s user + 0.000000s system = 0.031250s CPU (143.2%) new implementation: 0.373160s wall, 0.375000s user + 0.000000s system = 0.375000s CPU (100.5%)

Benchmark Times using a new generator for each uuid (10K loops): old implementation: 1.168479s wall, 1.171875s user + 0.000000s system = 1.171875s CPU (100.3%) new implementation: 0.010272s wall, 0.015625s user + 0.000000s system = 0.015625s CPU (152.1%)

These results look odd to me. What code exactly is being tested?

I went over it a number of times to be sure, and stepped through with the debugger to make sure it was actually going into the right code paths. I'll have to recreate the test as a separate benchmark that I can submit and build as part of the project. I was using boost::timer::auto_cpu_timer to do the timing. The "old implementation" is the current boost 1.65.1 uuids::random_generator. The "new implementation" would be the header-only random_device from my Boost.Random PR wrapped inside uuids::basic_random_generator, so there would be no seed() method and it would acquire 4 bytes of entropy at a time. That's why I wrote a separate test to acquire 16 bytes of entropy at a time, and that too was slower than reusing mt19937 for the same number of loops to generate uuids. When I have a PR to reference, I'll post it. - Jim

Peter Dimov

4:55 a.m.

James E. King, III wrote:

...

On Sat, Nov 4, 2017 at 10:59 PM, Peter Dimov via Boost wrote:

...
James E. King, III wrote:

Benchmark Times reusing a generator (1M loops):

...
old implementation: 0.021822s wall, 0.031250s user + 0.000000s system = 0.031250s CPU (143.2%) new implementation: 0.373160s wall, 0.375000s user + 0.000000s system = 0.375000s CPU (100.5%)

Benchmark Times using a new generator for each uuid (10K loops): old implementation: 1.168479s wall, 1.171875s user + 0.000000s system = 1.171875s CPU (100.3%) new implementation: 0.010272s wall, 0.015625s user + 0.000000s system = 0.015625s CPU (152.1%)

These results look odd to me. What code exactly is being tested?

I went over it a number of times to be sure, and stepped through with the debugger to make sure it was actually going into the right code paths. I'll have to recreate the test as a separate benchmark that I can submit and build as part of the project.

I was confused by the first set using 1M loops, and the second one using only 10K. Didn't catch that at first, so the results didn't make any sense. Need to multiply the second test by 100 for the two to be comparable. Have you tried RtlGenRandom by any chance?

James E. King, III

5:40 a.m.

On Sun, Nov 5, 2017 at 12:55 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...

James E. King, III wrote:

...
On Sat, Nov 4, 2017 at 10:59 PM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...
James E. King, III wrote:

Benchmark Times reusing a generator (1M loops):

...
old implementation: 0.021822s wall, 0.031250s user + 0.000000s system = 0.031250s CPU >> (143.2%) new implementation: 0.373160s wall, 0.375000s user + 0.000000s system = 0.375000s CPU >> (100.5%)

Benchmark Times using a new generator for each uuid (10K loops): old implementation: 1.168479s wall, 1.171875s user + 0.000000s system = 1.171875s CPU >> (100.3%) new implementation: 0.010272s wall, 0.015625s user + 0.000000s system = 0.015625s CPU >> (152.1%)

These results look odd to me. What code exactly is being tested?

I went over it a number of times to be sure, and stepped through with the debugger to make sure it was actually going into the right code paths. I'll have to recreate the test as a separate benchmark that I can submit and build as part of the project.

I was confused by the first set using 1M loops, and the second one using only 10K. Didn't catch that at first, so the results didn't make any sense. Need to multiply the second test by 100 for the two to be comparable.

Have you tried RtlGenRandom by any chance?

You cannot compare the 1M and 10K loops with much meaning: In the 1M loop the generator is constructed once then operator() is called 1M times. In the 10K loop the generator is constructed on each loop and operator() is called one time. The test would take way too long to be reasonable if I ran the latter 1M times. The intention is to show that seeding the mersenne twister (default random_generator) is expensive so you do it once, then it is relatively cheap to make random uuids from it. Conversely, bypassing the PRNG and using a random_device directly is quite slow when you go through wincrypt. In other words, the existing implementation may be optimal using a PRNG. I have not tried RtlGenRandom yet, but I am worried about removing Wincrypt in favor of that because it will break any implementation that uses its own entropy provider, for example a wincrypt driver that hooks up to a hardware entropy generator based on project requirements (government, etc). It's worth testing how fast it is however, so I will add that to the bench test I am putting together. I also have not tried the same on a POSIX system (urandom). - Jim

Peter Dimov

1 p.m.

James E. King, III wrote:

...

In other words, the existing implementation may be optimal using a PRNG.

This depends on your definition of optimality. If speed is more important than not having UUID collisions, sure, it's optimal.

...

I have not tried RtlGenRandom yet, but I am worried about removing Wincrypt in favor of that because it will break any implementation that uses its own entropy provider, for example a wincrypt driver that hooks up to a hardware entropy generator based on project requirements (government, etc).

What definition of "break" do you use here? I can't imagine that seeding a Mersenne twister with whatever hardware source of entropy can meet any government requirements, or that such a project will not generate its UUIDs directly from the entropy source, instead of going through a third-party library that needs to be audited and re-audited if upgraded. Plus, the entropy provider is hardcoded at present. What specific scenario will "break"?

Peter Dimov

1:04 p.m.

...

James E. King, III wrote:

...
In other words, the existing implementation may be optimal using a PRNG.

This depends on your definition of optimality. If speed is more important than not having UUID collisions...

... and predictability...

...

..., sure, it's optimal.

Andrey Semashev

1:17 p.m.

On 11/05/17 16:00, Peter Dimov via Boost wrote:

...

James E. King, III wrote:

...
In other words, the existing implementation may be optimal using a PRNG.

This depends on your definition of optimality. If speed is more important than not having UUID collisions, sure, it's optimal.

Mersenne twister has a very long period - probably longer than any given application can reasonably run to generate UUIDs, let alone a single random_generator instance. To eliminate even the slightest possibility of UUID collisions, you can re-seed the Mersenne twister PRNG periodically. And performance of generating UUIDs can be significant in some circumstances. UUIDs are typically used as various resource identifiers (think sessions, users, etc.), so poor performance can limit the rate of allocation of such resources.

Peter Dimov

1:37 p.m.

Andrey Semashev wrote:

...

On 11/05/17 16:00, Peter Dimov via Boost wrote:

...
James E. King, III wrote:

...
In other words, the existing implementation may be optimal using a PRNG.

This depends on your definition of optimality. If speed is more important than not having UUID collisions, sure, it's optimal.

Mersenne twister has a very long period - probably longer than any given application can reasonably run to generate UUIDs, let alone a single random_generator instance.

In general when random numbers are concerned, the best default option is to use crypto-secure numbers if you can afford it, because doing otherwise may have security implications. You could have an active attacker guessing the output of the PRNG given sufficient amount of output - which he'll have if you generate 50M UUIDs per second.

...

To eliminate even the slightest possibility of UUID collisions, you can re-seed the Mersenne twister PRNG periodically.

This will reduce or eliminate its performance advantage.

...

And performance of generating UUIDs can be significant in some circumstances. UUIDs are typically used as various resource identifiers (think sessions, users, etc.), so poor performance can limit the rate of allocation of such resources.

Sure. As I said, if speed is more important than quality, it's optimal to use a seeded-once Mersenne twister, and the option to do so will not be going away. When it isn't, it isn't. (I've trouble imagining a scenario in which you won't run out of resources if you allocate 50M resource identifiers per second, but let's suppose one exists, for the sake of argument.)

James E. King, III

2:11 p.m.

On Sun, Nov 5, 2017 at 8:37 AM, Peter Dimov via Boost

...

wrote:

...

Andrey Semashev wrote:

...
On 11/05/17 16:00, Peter Dimov via Boost wrote:

...
James E. King, III wrote:

...
In other words, the existing implementation may be optimal using a >> PRNG.

This depends on your definition of optimality. If speed is more > important than not having UUID collisions, sure, it's optimal.

Mersenne twister has a very long period - probably longer than any given application can reasonably run to generate UUIDs, let alone a single random_generator instance.

In general when random numbers are concerned, the best default option is to use crypto-secure numbers if you can afford it, because doing otherwise may have security implications. You could have an active attacker guessing the output of the PRNG given sufficient amount of output - which he'll have if you generate 50M UUIDs per second.

To eliminate even the slightest possibility of UUID collisions, you can

...
re-seed the Mersenne twister PRNG periodically.

This will reduce or eliminate its performance advantage.

And performance of generating UUIDs can be significant in some

...
circumstances. UUIDs are typically used as various resource identifiers (think sessions, users, etc.), so poor performance can limit the rate of allocation of such resources.

Sure. As I said, if speed is more important than quality, it's optimal to use a seeded-once Mersenne twister, and the option to do so will not be going away. When it isn't, it isn't.

(I've trouble imagining a scenario in which you won't run out of resources if you allocate 50M resource identifiers per second, but let's suppose one exists, for the sake of argument.)

I'm adding boost::uuids::random_device_solo which is a basic_random_generator of random_device (header-only) type, and documented it as optimal for one-off UUID generation, whereas the standard random_generator is best suited to be reused (in a thread-safe manner) for multiple uuid generation. boost::uuids::uuid u = boost::uuids::random_generator_solo()(); // avoids PRNG setup overhead I'm also adding unit tests to properly test all the Wincrypt and Bcrypt failure paths in the header-only random_device. It won't show up in the codecov reports from the linux build however. It would be nice to have a more automated integration for leveraging generation of gmocks inside Boost.WinAPI somehow... (I'm not using gmock for this effort, just a suggestion) it would make mocking the Win32 APIs easier. - Jim

Peter Dimov

2:19 p.m.

James E. King, III wrote:

...

I'm adding boost::uuids::random_device_solo which is a basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

...

... and documented it as optimal for one-off UUID generation, whereas the standard random_generator is best suited to be reused (in a thread-safe manner) for multiple uuid generation.

You're still using "optimal" only with respect to performance. It's never optimal for random numbers to not be cryptographically secure.

James E. King, III

2:45 p.m.

On Sun, Nov 5, 2017 at 9:19 AM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

I'm adding boost::uuids::random_device_solo which is a

...
basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

It's not committed yet. :>

...

... and documented it as optimal for one-off UUID generation, whereas the

...
standard random_generator is best suited to be reused (in a thread-safe manner) for multiple uuid generation.

You're still using "optimal" only with respect to performance. It's never optimal for random numbers to not be cryptographically secure.

Issues are always welcome. - Jim

Peter Dimov

3:20 p.m.

James E. King, III wrote:

...

...
You're still using "optimal" only with respect to performance. It's never optimal for random numbers to not be cryptographically secure.

Issues are always welcome.

Your use of "optimal" shows that you do not agree with the principle that generated random numbers should always be of cryptographic quality by default unless explicitly chosen by the user to not be. This fundamental mindset difference is not something issues can fix.

...

...
...
I'm adding boost::uuids::random_device_solo which is a basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

It's not committed yet. :>

The logical choice would be random_device_generator but I don't like it because you'll give it a "token" parameter and insist on passing it to Wincrypt because unspecified multinationals using XP and a hardware entropy source will need it to meet hypothetical government standards.

James E. King, III

4:23 p.m.

On Sun, Nov 5, 2017 at 10:20 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...

James E. King, III wrote:

...
You're still using "optimal" only with respect to performance. It's > never optimal for random numbers to not be cryptographically secure.

Issues are always welcome.

Your use of "optimal" shows that you do not agree with the principle that generated random numbers should always be of cryptographic quality by default unless explicitly chosen by the user to not be. This fundamental mindset difference is not something issues can fix.

Actually the patient is open for surgery so I am quite open to making changes. You are suggesting that: 1. The default random_generator use a random_device for entropy. This is a breaking change, as the explicit constructors for random_generator would no longer take a boost::mt19937, so if anyone has this code they would need to change their random_generator to something else. Perhaps provide a random_generator_compat for them? 2. My use of optimal was relative to performance, yes. That does not mean what I said is what's best. That's why we're discussing it here, to reach alignment.

...

...
...
I'm adding boost::uuids::random_device_solo which is a > > basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

I changed it to random_generator_os, temporarily, but pending more discussion it may become random_generator.

...

It's not committed yet. :>

...
The logical choice would be random_device_generator but I don't like it because you'll give it a "token" parameter and insist on passing it to Wincrypt because unspecified multinationals using XP and a hardware entropy source will need it to meet hypothetical government standards.

I have already removed the token mechanism from the uuid detail implementation because it is unnecessary. Anyone is free to pass in their own UniformRandomNumberGenerator for which they can provide their own implementation, which will cover hardware-based entropy generation use cases. Boost.Uuid will provide a standard header-only random_device that uses bcrypt/cloudabi/file/wincrypt as necessary to meet platform requirements, but it will be in "detail". It is still a bit more complex than it needs to be so I am continuing to simplify it from what it was in the pull request for Boost.Random. For example I want to eliminate the variate/uniform_int generator layer and just use the UniformRandomNumberGenerators operator() directly. Again these are a work in progress and not committed yet. Happy to have picked up new tricks along the way, like using boost::enable_if and boost::tti to properly handle UniformRandomNumberGenerators that are PseudoRandomNumberGenerators depending on whether a seed method exists. - Jim

Andrey Semashev

7:33 p.m.

On 11/05/17 19:23, James E. King, III via Boost wrote:

...

On Sun, Nov 5, 2017 at 10:20 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...
James E. King, III wrote:

...
...
I'm adding boost::uuids::random_device_solo which is a > > basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

I changed it to random_generator_os, temporarily, but pending more discussion it may become random_generator.

I can throw in another candidate. :) Since the new generator is supposed to be using solely random entropy from system, I would suggest random_entropy_generator.

Paul A. Bristow

6 Nov 6 Nov

10:31 a.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev via Boost Sent: 05 November 2017 19:33 To: boost@lists.boost.org Cc: Andrey Semashev Subject: Re: [boost] Boost.Uuid and header-only support

On 11/05/17 19:23, James E. King, III via Boost wrote:

...
On Sun, Nov 5, 2017 at 10:20 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...
James E. King, III wrote:

...
...
I'm adding boost::uuids::random_device_solo which is a > > basic_random_generator of random_device (header-only) type, ...

That's not a very good name. :-)

I changed it to random_generator_os, temporarily, but pending more discussion it may become random_generator.

I can throw in another candidate. :) Since the new generator is supposed to be using solely random entropy from system, I would suggest random_entropy_generator.

I have been following this erudite discussion with interest, but ignorance. Without wishing to add even more entropy, may I suggest that adding more options is rarely a bad thing, but it creates an even more bewildering set of options for the user (for UUID, I fear as ignorant as I). It would be very helpful if a summary of the pros and cons of generators was added to the documentation along with several examples of good use. Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 (0) 1539 561830

Peter Dimov

12:25 p.m.

Paul A. Bristow wrote:

...

I have been following this erudite discussion with interest, but ignorance.

Without wishing to add even more entropy, may I suggest that adding more options is rarely a bad thing, but it creates an even more bewildering set of options for the user (for UUID, I fear as ignorant as I).

That is why I'd like the default generator to do the right thing and doing the wrong thing to be harder and not be presented as an equivalent alternative.

Peter Dimov

12:49 p.m.

...

Paul A. Bristow wrote:

...
I have been following this erudite discussion with interest, but ignorance.

Without wishing to add even more entropy, may I suggest that adding more options is rarely a bad thing, but it creates an even more bewildering set of options for the user (for UUID, I fear as ignorant as I).

That is why I'd like the default generator to do the right thing and doing the wrong thing to be harder and not be presented as an equivalent...

... or more "optimal"...

...

alternative.

James E. King, III

1:26 p.m.

On Mon, Nov 6, 2017 at 7:49 AM, Peter Dimov via Boost

...

wrote:

...

Paul A. Bristow wrote:

...
...
I have been following this erudite discussion with interest, but > ignorance.

Without wishing to add even more entropy, may I suggest that adding more > options is rarely a bad thing, but it creates an even more bewildering > set of options for the user (for UUID, I fear as ignorant as I).

That is why I'd like the default generator to do the right thing and doing the wrong thing to be harder and not be presented as an equivalent...

... or more "optimal"...

alternative.

...

I'm planning to change random_generator to use the header-only random_device based on points made in this discussion: 1. It is more secure. 2. I believe it is the most widely used use case: generating a uuid relatively infrequently (i.e. not in bulk). Therefore it will look something like this: // // The default random_generator uses operating-system provided entropy, // is the most secure, and fastest random uuid generator for creating a // small number of uuids with a single generator because it does not need // expensive seeding to be effective like a PseudoRandomNumberGenerator // does. // typedef basic_random_generatordetail::random::random_device random_generator; // // The bulk generator uses a mersenne twister to make random uuid generation of // many uuids from a single generator more efficient, at the expense of security // and guaranteed randomness. // typedef basic_random_generator<mt19937> random_generator_bulk; This is a breaking change for anyone providing their own mersenne twister to the random_generator constructors that take a UniformRandomNumberGenerator. I expect this would be quite rare, and in that case one could switch to using _bulk easily enough. Please correct me if I misinterpreted anything. I'll have a PR into my fork for a dry-run through CI later today and I'll post the link here. - Jim

Peter Dimov

1:38 p.m.

James E. King, III wrote:

...

I'm planning to change random_generator to use the header-only random_device based on points made in this discussion:

1. It is more secure. 2. I believe it is the most widely used use case: generating a uuid relatively infrequently (i.e. not in bulk).

Thanks.

...

Therefore it will look something like this:

// // The default random_generator uses operating-system provided entropy, // is the most secure, and fastest random uuid generator for creating a // small number of uuids with a single generator because it does not need // expensive seeding to be effective like a PseudoRandomNumberGenerator // does. // typedef basic_random_generatordetail::random::random_device random_generator;

I suggest class random_generator { public: typedef uuid result_type; random_generator(); uuid operator()(); }; that is, not using basic_random_generator here at all. There is no need to go through a URNG in this case; random bytes can and should be obtained directly from the OS API.

...

typedef basic_random_generator<mt19937> random_generator_bulk;

If we guarantee that this will always use mt19937, it might be better to just call it random_generator_mt19937.

James E. King, III

2:21 p.m.

On Mon, Nov 6, 2017 at 8:38 AM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

...
I'm planning to change random_generator to use the header-only random_device based on points made in this discussion:

1. It is more secure. 2. I believe it is the most widely used use case: generating a uuid relatively infrequently (i.e. not in bulk).

Thanks.

Therefore it will look something like this:

...
// // The default random_generator uses operating-system provided entropy, // is the most secure, and fastest random uuid generator for creating a // small number of uuids with a single generator because it does not need // expensive seeding to be effective like a PseudoRandomNumberGenerator // does. // typedef basic_random_generatordetail::random::random_device random_generator;

I suggest

class random_generator { public: typedef uuid result_type;

random_generator();

uuid operator()(); };

that is, not using basic_random_generator here at all. There is no need to go through a URNG in this case; random bytes can and should be obtained directly from the OS API.

typedef basic_random_generator<mt19937> random_generator_bulk;

...
If we guarantee that this will always use mt19937, it might be better to just call it random_generator_mt19937.

And I just wrapped up my changes with random_generator_bulk too. I'm not certain that it is necessary to make random_generator a separate class like this. As it is, I have removed most of the additional weight from basic_random_generator like the variate and uniform code. I'll turn the crank one more time however to see if I can make any further improvements. - Jim

Peter Dimov

2:34 p.m.

James E. King, III wrote:

...

I'm not certain that it is necessary to make random_generator a separate class like this.

If it's documented as an instance of basic_random_generator<>, making it later not an instance would be a breaking change. It's also possible to document it as typedef /*unspecified*/ random_generator; but then we need to specify its interface by basically repeating that same class definition.

Steven Watanabe

4:10 p.m.

AMDG On 11/06/2017 07:21 AM, James E. King, III via Boost wrote:

...

I'm not certain that it is necessary to make random_generator a separate class like this. As it is, I have removed most of the additional weight from basic_random_generator like the variate and uniform code. I'll turn the crank one more time however to see if I can make any further improvements.

uniform_int_distribution is not excess weight. You could use independent_bits_engine instead, but if you accept an arbitrary random engine, then you need some way to convert its output into the correct range. Not all engines produce uniformly distributed 32-bit output. In Christ, Steven Watanabe

James E. King, III

5:20 p.m.

On Mon, Nov 6, 2017 at 11:10 AM, Steven Watanabe via Boost < boost@lists.boost.org> wrote:

...

uniform_int_distribution is not excess weight. You could use independent_bits_engine instead, but if you accept an arbitrary random engine, then you need some way to convert its output into the correct range. Not all engines produce uniformly distributed 32-bit output.

Filling a uuid with random does not require integers. In this case as long as the UniformRandomNumberGenerator produces at least one byte from operator() then it is sufficient to fill the contents of a uuid randomly. If it returns more than that, it will be more efficient. I have a macro that makes a header-only random_device type and it is used as follows: namespace detail { BOOST_TTI_HAS_MEMBER_FUNCTION(seed) } template<class MaybePseudoRandomNumberGenerator> typename boost::enable_if >::type seed(MaybePseudoRandomNumberGenerator& rng) { BOOST_UUID_DETAIL_MAKE_RANDOM_DEVICE_TYPE(unsigned int, seed_random_device); using boost::function_input_iterator; using boost::make_function_input_iterator; boost::infinite inf; seed_random_device egen; function_input_iterator beg = make_function_input_iterator(egen, inf); function_input_iterator end = make_function_input_iterator(egen, inf); rng.seed(beg, end); } internally this is used to obtain seeding for PRNGs. One can use this in various ways, and it is not limited to the unsigned int size, for example: BOOST_UUID_DETAIL_MAKE_RANDOM_DEVICE_TYPE(uuid, uuid_random_device); typedef basic_random_generator random_generator; Now the random_generator obtains 16 bytes of entropy in a single call, rather than making 4 calls for 4 bytes each. This technique might be applicable to the existing PRNGs in Boost.Random with a header-only random_device implementation. For example it may be possible to have the mersenne twister make one entropy call for 623 bytes instead of making 156 calls for 4 bytes at a time. I haven't looked in detail at the way it seeds itself, however: struct mersenne_entropy_pod { uint8_t entropy[623]; } BOOST_UUID_DETAIL_MAKE_RANDOM_DEVICE_TYPE(mersenne_entropy_pod, mersenne_seed_random_device); Then mersenne_twister could call mersenne_seed_random_device()() to get 623 bytes of entropy in a single system call. - Jim

Peter Dimov

6:05 p.m.

James E. King, III wrote:

...

Filling a uuid with random does not require integers. In this case as long as the UniformRandomNumberGenerator produces at least one byte from operator() then it is sufficient to fill the contents of a uuid randomly.

If you just take the lowest 8 bits of the output, it will not be uniformly distributed unless min() == 0 and max() == 2^n-1.

James E. King, III

10:50 p.m.

On Mon, Nov 6, 2017 at 1:05 PM, Peter Dimov via Boost

...

wrote:

...

James E. King, III wrote:

Filling a uuid with random does not require integers.

...
In this case as long as the UniformRandomNumberGenerator produces at least one byte from operator() then it is sufficient to fill the contents of a uuid randomly.

If you just take the lowest 8 bits of the output, it will not be uniformly distributed unless min() == 0 and max() == 2^n-1.

I submitted a pull request so anyone who wants to can take a look. https://github.com/boostorg/uuid/pull/52 - Jim

Andrey Semashev

2:35 p.m.

On 11/06/17 16:26, James E. King, III via Boost wrote:

...

On Mon, Nov 6, 2017 at 7:49 AM, Peter Dimov via Boost
...
wrote:

...
Paul A. Bristow wrote:

...
...
I have been following this erudite discussion with interest, but > ignorance.

Without wishing to add even more entropy, may I suggest that adding more > options is rarely a bad thing, but it creates an even more bewildering > set of options for the user (for UUID, I fear as ignorant as I).

That is why I'd like the default generator to do the right thing and doing the wrong thing to be harder and not be presented as an equivalent...

... or more "optimal"...

alternative.

...
I'm planning to change random_generator to use the header-only random_device based on points made in this discussion:

1. It is more secure. 2. I believe it is the most widely used use case: generating a uuid relatively infrequently (i.e. not in bulk).

I still think that having random_generator as an object is an advantage that people probably use for performance. I would prefer the entropy-based generator to be provided under a different new name. I understand the arguments for more secure implementation by default, although it appears to me that after the change the more secure implementation is going to be the only one provided by Boost.UUID, not just the default. What I disagree with is that the cost of more security is going to be imposed on all users, whether they need it or not. And frankly, I don't think that security should depend on UUID properties in the first place - UUIDs are *not* a security measure, it is a piece of data that should be adequately protected by a separate security layer. If you don't protect it and rely on its properties like unpredictability then you're wrong. The primary use of UUIDs is an id - a slightly better version of a counter that doesn't need to be synchronized between threads or processes. Name-based generators also allow it to be a sort of a hash function as well. Thus it should be cheap to create and operate on. Having a more secure random generator may be useful as an option, but really the primary concern about UUIDs is their uniqueness, not predictability. An attacker should not have an access to your sensitive data, including UUIDs, in the first place.

James E. King, III

3:31 p.m.

On Mon, Nov 6, 2017 at 9:35 AM, Andrey Semashev via Boost < boost@lists.boost.org> wrote:

...

On 11/06/17 16:26, James E. King, III via Boost wrote:

...
On Mon, Nov 6, 2017 at 7:49 AM, Peter Dimov via Boost < boost@lists.boost.org

...
wrote:

Paul A. Bristow wrote:

...
...
I have been following this erudite discussion with interest, but >

...
ignorance.

...
Without wishing to add even more entropy, may I suggest that adding

more > options is rarely a bad thing, but it creates an even more bewildering > set of options for the user (for UUID, I fear as ignorant as I).

That is why I'd like the default generator to do the right thing and doing the wrong thing to be harder and not be presented as an equivalent...

... or more "optimal"...

alternative.

...
I'm planning to change random_generator to use the header-only

random_device based on points made in this discussion:

1. It is more secure. 2. I believe it is the most widely used use case: generating a uuid relatively infrequently (i.e. not in bulk).

I still think that having random_generator as an object is an advantage that people probably use for performance. I would prefer the entropy-based generator to be provided under a different new name.

I understand the arguments for more secure implementation by default, although it appears to me that after the change the more secure implementation is going to be the only one provided by Boost.UUID, not just the default. What I disagree with is that the cost of more security is going to be imposed on all users, whether they need it or not. And frankly, I don't think that security should depend on UUID properties in the first place - UUIDs are *not* a security measure, it is a piece of data that should be adequately protected by a separate security layer. If you don't protect it and rely on its properties like unpredictability then you're wrong.

The primary use of UUIDs is an id - a slightly better version of a counter that doesn't need to be synchronized between threads or processes. Name-based generators also allow it to be a sort of a hash function as well. Thus it should be cheap to create and operate on. Having a more secure random generator may be useful as an option, but really the primary concern about UUIDs is their uniqueness, not predictability. An attacker should not have an access to your sensitive data, including UUIDs, in the first place.

Based on testing, it is faster to use the non-deterministic UniformRandomNumberGenerator provided by the operating system if you are creating less than 100 uuids (on my system, obviously). I believe the majority of users are probably calling boost::uuids::random_generator()() to make a uuid as the documentation had nothing in it about performance. It does now, at least in my sandbox. Once the PR is ready for review I will post it here for review. Peter's last email led me to change how random_device is defined so now it is quite easy to make an operating system provided random_device typedef with a configurable entropy size using a single macro. Therefore where a PRNG is used and it wants to seed with an iterator taking unsigned int, and in the case where we want 16 bytes in a single call from the OS, both cases are now optimal. - Jim

Niall Douglas

5 Nov 5 Nov

2:08 p.m.

...

So the initialization of the mersenne twister is expensive, so if you do it once and then reuse it, it is much more efficient than going to Wincrypt.

Mersenne is only fast on x86/x64. It is dog slow on every other CPU. Also, I hate to be pedantic, but Mersenne is not cryptographically secure. UUIDs ought to be generated from a crypto-strong source unless the end user specifically asks for otherwise. On Windows, that's RtlGenRandom() which is rand_s() in recent MSVC's. On POSIX that is /dev/urandom or /dev/random depending on the POSIX flavour. If you want something faster, Blake2b is crypto strong and fast on all modern CPUs. Just need to seed it with a random seed, and you're happy. Failing that, really recent CPUs can do SHA256 in hardware, but they have tiny market penetration currently. You can, of course, use any AES hardware offload also as a randomness generator, that has much wider support in CPUs. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

James E. King, III

6 Nov 6 Nov

1:17 p.m.

On Sun, Nov 5, 2017 at 9:08 AM, Niall Douglas via Boost < boost@lists.boost.org> wrote:

...

...
So the initialization of the mersenne twister is expensive, so if you do it once and then reuse it, it is much more efficient than going to Wincrypt.

Mersenne is only fast on x86/x64. It is dog slow on every other CPU.

Also, I hate to be pedantic, but Mersenne is not cryptographically secure. UUIDs ought to be generated from a crypto-strong source unless the end user specifically asks for otherwise.

On Windows, that's RtlGenRandom() which is rand_s() in recent MSVC's. On POSIX that is /dev/urandom or /dev/random depending on the POSIX flavour.

If you want something faster, Blake2b is crypto strong and fast on all modern CPUs. Just need to seed it with a random seed, and you're happy. Failing that, really recent CPUs can do SHA256 in hardware, but they have tiny market penetration currently. You can, of course, use any AES hardware offload also as a randomness generator, that has much wider support in CPUs.

Niall

Is there an existing PRNG in Boost.Random that would be better than mersenne to use in a "bulk" uuid generator typedef?

Steven Watanabe

3:57 p.m.

AMDG On 11/06/2017 06:17 AM, James E. King, III via Boost wrote:

...

On Sun, Nov 5, 2017 at 9:08 AM, Niall Douglas via Boost < boost@lists.boost.org> wrote:

...
...
So the initialization of the mersenne twister is expensive, so if you do it once and then reuse it, it is much more efficient than going to Wincrypt.

Mersenne is only fast on x86/x64. It is dog slow on every other CPU.

Also, I hate to be pedantic, but Mersenne is not cryptographically secure. UUIDs ought to be generated from a crypto-strong source unless the end user specifically asks for otherwise.

On Windows, that's RtlGenRandom() which is rand_s() in recent MSVC's. On POSIX that is /dev/urandom or /dev/random depending on the POSIX flavour.

If you want something faster, Blake2b is crypto strong and fast on all modern CPUs. Just need to seed it with a random seed, and you're happy. Failing that, really recent CPUs can do SHA256 in hardware, but they have tiny market penetration currently. You can, of course, use any AES hardware offload also as a randomness generator, that has much wider support in CPUs.

Niall

Is there an existing PRNG in Boost.Random that would be better than mersenne to use in a "bulk" uuid generator typedef?

Boost.Random does not have any cryptographic prngs (apart from random_device). In Christ, Steven Watanabe

Peter Dimov

8 Nov 8 Nov

1:45 a.m.

Steven Watanabe wrote:

...

On 11/06/2017 06:17 AM, James E. King, III via Boost wrote:

...
Is there an existing PRNG in Boost.Random that would be better than mersenne to use in a "bulk" uuid generator typedef?

Boost.Random does not have any cryptographic prngs (apart from random_device).

I came across https://gist.github.com/orlp/32f5d1b631ab092608b1 in the course of reading http://www.pcg-random.org/posts/on-trivial-predictability.html ChaCha20 looks like a reasonable candidate for Boost.Random, and the license of Orson Peters's implementation seems Boost-compatible.

Niall Douglas

6 Nov 6 Nov

8:02 p.m.

...

...
If you want something faster, Blake2b is crypto strong and fast on all modern CPUs. Just need to seed it with a random seed, and you're happy. Failing that, really recent CPUs can do SHA256 in hardware, but they have tiny market penetration currently. You can, of course, use any AES hardware offload also as a randomness generator, that has much wider support in CPUs.

Is there an existing PRNG in Boost.Random that would be better than mersenne to use in a "bulk" uuid generator typedef?

No idea. I avoid Boost dependencies when I can, and Blake2b source code is well written. Note that probably the hardware randomness generator instruction RDRAND in Intel CPUs will be sufficient for UUID generation. It passes all the tests for crypto strength, but a lot of people don't trust Intel. Many ARM chips also have hardware randomness generator offload. I forget the coprocessor instruction. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

James E. King, III

4 Nov 4 Nov

5:37 p.m.

On Sat, Nov 4, 2017 at 11:40 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:

...

Andrey Semashev wrote:

The proposal to make Boost.Random header-only was rejected by Steven.

...
https://github.com/boostorg/random/pull/29

This sounds like essential context that might have been worth mentioning. :-)

I specifically did not mention the PR resolution because my intention was to find a path forward, not to bring attention to that decision. Steven is right that we're still dependent on Boost.System, this needs to

...

be solved somehow.

To get back to Uuid. The whole seed_rng file needs to be retired. It's only an implementation detail of a primitive operation that returns N random bytes. At the time seed_rng was written, there was no good way of obtaining high quality randomness from the OS, but this is not the case today.

boost::uuids::detail::seed_rng has the following problems: 1. It duplicates the behavior of boost::random_device. 2. It is unnecessarily complex (does way more than boost::random_device). 3. It ignores errors, leading potentially to not properly seeding the RNG. 4. It has no support for Universal Windows Platform.

...

What the Uuid library needs, when generating a random UUID, is just an array of random bytes; there's no need to create a random number generator, then seed it. None at all. This is historical baggage.

And in my opinion, all the conceptification and templatization in James's PR is completely unnecessary and just adds complexity. What we need is this:

namespace boost

...

{ int get_random_bytes( void * buffer, size_t size ); // returns errno }

This happens to match the Linux function with the same name but this is coincidental.

The implementation of this function needs to

- read from /dev/urandom on POSIX - use RtlGenRandom on Windows desktop - use Bcrypt on Windows non-desktop

It would be safer to use Wincrypt on windows desktop for these reasons: 1. RtlGenRandom has no published interface and may disappear at any time. 2. Consumers who use their own Wincrypt provider for entropy would be broken by such a change.

...

We have to decide where to put it though. Utility springs to mind as a first candidate.

Something called "get_random_bytes" seems like it would belong in Boost.Random...

...

Switching topics back to Boost.Random, I consider the use of the token to select a cryptographic provider a bad practice. Even the POSIX path should ignore the token. get_random_bytes is the way to go. But that's a separate story.

The token allows the POSIX implementation to use a different entropy provider device (other than /dev/urandom) or even a text file. There may be reasons why folks use their own, for example in the government sector, so any change in that area would be a breaking change. Thanks, - Jim

Peter Dimov

5:46 p.m.

James E. King, III wrote:

...

It would be safer to use Wincrypt on windows desktop for these reasons:

1. RtlGenRandom has no published interface...

https://msdn.microsoft.com/en-us/library/windows/desktop/aa387694(v=vs.85).a...

...

and may disappear at any time.

If it does, we'll deal with it.

...

2. Consumers who use their own Wincrypt provider for entropy would be broken by such a change.

First, there are no such consumers, and second, get_random_bytes has no way to supply a "provider", by design.

...

The token allows the POSIX implementation to use a different entropy provider device (other than /dev/urandom)...

I know. This is wrong and should never be used.

...

or even a text file.

random_device is not the correct or portable way to use a text file as a source of random numbers.

Glen Fernandes

12:42 a.m.

On Fri, Nov 3, 2017 at 7:58 PM, James E. King, III wrote:

...

I have an open defect related to a random seed generator that's currently in boost::uuids::detail called seed_rng. The class does a fair bit more than it really needs to, and does not support Universal Windows Platform (UWP) store targets. It is otherwise quite similar to boost::random::random_device however it is a header-only implementation (in fact seed_rng has a comment about putting it into Boost.Random). I was planning on removing it and leaving randomness to Boost.Random because that's where the code really belongs, however this would introduce a new library dependency on Boost.Random for any code using boost::uuids::random_generator. So I'm curious on how important folks believe it is to maintain the header-only status of Boost.Uuid.

While I don't use Boost.Uuid, I do think users of Boost enjoy the convenience of header-only libraries (and know of at some who are restricted to - within their organization - only the header-only libraries). Why not make Boost.Random header-only first, before taking the dependency on it? Glen

Andrey Semashev

11:39 a.m.

On 11/04/17 02:58, James E. King, III via Boost wrote:

...

Hi folks,

I have an open defect related to a random seed generator that's currently in boost::uuids::detail called seed_rng. The class does a fair bit more than it really needs to, and does not support Universal Windows Platform (UWP) store targets. It is otherwise quite similar to boost::random::random_device however it is a header-only implementation (in fact seed_rng has a comment about putting it into Boost.Random). I was planning on removing it and leaving randomness to Boost.Random because that's where the code really belongs, however this would introduce a new library dependency on Boost.Random for any code using boost::uuids::random_generator. So I'm curious on how important folks believe it is to maintain the header-only status of Boost.Uuid.

I've expressed my opinion to you privately and will reproduce it here just for the record. I think the header-only property of Boost.UUID is an important feature that enables some users to use the library. Whether that is due to a project/company policy, pure convenience or other reasons is irrelevant; the important part is that some users simply won't use separately compiled libraries. Transiting to non-header-only state puts those users in an awkward position, having to either port away from Boost.UUID (likely, to a home-grown solution) or to adjust their policies and build environment. Neither of those are good and easy outcomes. Depending on how exactly you want to implement this, adding linking dependency on Boost.Random can break POSIX users as autolinking is not available on POSIX. (I.e. POSIX can be saved if you keep the current header-only code for POSIX and only limit your change to Windows.) Although autolinking can somewhat hide the problem on Windows, I don't consider it a solution because it won't help if the compiled libraries are simply not built. With the Boost.Random proposal rejected, I can see the following possible solutions: 1. Add the bcrypt-based implementation to the existing ones in seed_rng.hpp. I think, this is the simplest to implement and it will keep the users happy. Personally, I would do this. 2. Add your header-only random_device implementation proposed for Boost.Random as an implementation detail to Boost.UUID and use it internally. 3. Use std::random on newer compilers on Windows. Arguably, std::random_device is supported on UWP as well as other Windows flavors. Keep the current code for all other platforms and older compilers on Windows. 4. Use std::random universally on C++11 and later and Boost.Random on C++03. This will break C++03 users but will maintain the header-only property for C++11 and on. Personally, I don't see the point in forcing the use of std::random or Boost.Random. It looks nice from the maintainer's point of view (i.e. separate concerns and let the random library deal with RNG), but given that most of the code is already written in Boost.UUID and forcing Boost.Random is actively harmful to Boost.UUID it is not worth it.

2575

Age (days ago)

2580

Last active (days ago)

List overview

Download

48 comments

8 participants

participants (8)

Andrey Semashev
Edward Diener
Glen Fernandes
James E. King, III
Niall Douglas
Paul A. Bristow
Peter Dimov
Steven Watanabe