boost::random, header-only, boost::uuids::detail::seed_rng, and UWP development

newer
Searching for Review Manager for...

James E. King, III

12 Sep 2017 12 Sep '17

12:40 p.m.

Hi folks, It looks like the only reason boostorg/random is not a header-only library is due to random_device (it is the only src/ file: https://github.com/boostorg/random/tree/develop/src). The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community. In the boostorg/uuid library there is a seed_rng class which is doing similar things (however providing more entropy) and the library is header-only. In addition there are two filed defects - one in Trac and one in github issues, about uuid being incompatible with UWP applications (that's Windows universal apps). In uuid, I started adding support for BCrypt which is a Windows API supported on later versions of Windows, and once I changed my build flags to target Windows 10 and WINAPI_FAMILY_APP, I started getting build failures in random_device. On further inspection it looks like the random library doesn't have support for BCrypt yet, and even worse as a source library one would need to build two variants - one for WinCrypt and one for BCrypt to support older and newer windows platforms. There are also some implications on MinGW, as the older original MinGW has support for WinCrypt and MinGW-64 supports both. I'm going to propose that the implementation from boost::uuids::detail::seed_rng be moved into boost::random as the new random_device implementation. At the same time, boost::random will become header-only, and thus support setting _WIN32_WINNT at build time of consuming applications properly, and therefore support new Windows platforms properly. At the same time I would like to change seed_rng such that it no longer silently ignores some return values, and does not disable any compiler warnings (deprecations, currently). Thoughts? Thanks, - Jim

Show replies by date

degski

12 Sep 12 Sep

1:27 p.m.

On 12 September 2017 at 15:40, James E. King, III via Boost < boost@lists.boost.org> wrote:

...

The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

You worry too much, just call rand_s https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx (the same thing under the hood, but taking care of any WinCrypt BCrypt, whatever issues). In the boostorg/uuid library there is a seed_rng class which is doing

...

similar things (however providing more entropy) and the library is header-only.

The rand_s function uses the operating system to generate cryptographically secure random numbers. It doesn't get better than that, maybe a call to RdRand https://en.wikipedia.org/wiki/RdRand#RDSEED on a current Intel CPU, which would use thermal noise (but you'll have to take Intel's word for it and just cross your fingers they didn't sell you out to a 3-letter agency). degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

Andrey Semashev

2:13 p.m.

On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost wrote:

...

On 12 September 2017 at 15:40, James E. King, III via Boost < boost@lists.boost.org> wrote:

...
The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

You worry too much, just call rand_s https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx (the same thing under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

I assume, this is MSVC-specific, i.e. doesn't work with MinGW?

James E. King, III

2:34 p.m.

Part of any effort here would also include taking the Travis and Appveyor CI files from boost::uuid and run all the checks in boost::random therefore any incompatibilities such as the one expressed for MinGW would be identified. In fact, it would make sense to add some additional unit test files and/or builds to Appveyor to check different _WIN32_WINNT compatibility levels to ensure all the build-time code branches are covered (as well as WINAPI_FAMILY differences). The documentation on rand_s doesn't lead me to believe it is compatible with all platforms, as it expressly states it depends on an API that only works on Windows Desktop applications, however a well formed CI environment would prove it out if rand_s was to be used. - Jim On Tue, Sep 12, 2017 at 10:13 AM, Andrey Semashev via Boost < boost@lists.boost.org> wrote:

...

On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost wrote:

...
On 12 September 2017 at 15:40, James E. King, III via Boost < boost@lists.boost.org> wrote:

...
The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

You worry too much, just call rand_s https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx (the same thing under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

I assume, this is MSVC-specific, i.e. doesn't work with MinGW?

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/ mailman/listinfo.cgi/boost

Peter Dimov

2:35 p.m.

Andrey Semashev wrote:

...

On Tue, Sep 12, 2017 at 4:27 PM, degski via Boost wrote:

...
On 12 September 2017 at 15:40, James E. King, III via Boost < boost@lists.boost.org> wrote:

...
The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

You worry too much, just call rand_s https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx (the same thing under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

I assume, this is MSVC-specific, i.e. doesn't work with MinGW?

In addition, "The rand_s function requires that constant _CRT_RAND_S be defined prior to the inclusion statement for the function to be declared, as in the following example: #define _CRT_RAND_S #include " which probably means that if is included prior to the header that wants to use rand_s, it will be unavailable. We could use RtlGenRandom instead, which is what rand_s calls. Not sure if that works under UWP. To answer the question in the original post, yes, making random_device header-only would indeed be useful, as long as we don't include . There's no need to include the implementation from seed_rng in its entirety though, it's more complicated than need be because it tries to support the case in which a device isn't available. Since the construction of random_device can fail, it can be a lightweight wrapper over /dev/urandom / CryptGenRandom, without all the SHA business.

degski

2:56 p.m.

On 12 September 2017 at 17:35, Peter Dimov via Boost wrote:

...

... making random_device header-only would indeed be useful, as long as we don't include .

The below works just fine for me, no header: #include <iostream> extern "C" void rand_s ( unsigned int* ); int main ( ) { unsigned int x; rand_s ( &x ); std::cout << x << '\n'; return 0; } Isn't "C" great (sometimes). degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

Peter Dimov

3:14 p.m.

degski wrote:

...

The below works just fine for me, no header:

#include <iostream>

extern "C" void rand_s ( unsigned int* );

Something like that could probably work, but this is not the correct prototype. rand_s returns errno_t. If you try #define _CRT_RAND_S #include extern "C" void rand_s( unsigned int* ); you get an error. errno_t seems to be just a typedef for int though.

degski

3:38 p.m.

On 12 September 2017 at 18:14, Peter Dimov via Boost wrote:

...

Something like that could probably work, but this is not the correct prototype. rand_s returns errno_t. If you try

The code I showed compiles and works, it won't allow you to get the error-code, though... So it should (because you are right) be (if one is interested in an error code): extern "C" int rand_s ( unsigned int* ); degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

Edward Diener

13 Sep 13 Sep

10:21 p.m.

On 9/12/2017 9:27 AM, degski via Boost wrote:

...

On 12 September 2017 at 15:40, James E. King, III via Boost < boost@lists.boost.org> wrote:

...
The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

You worry too much, just call rand_s https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx (the same thing under the hood, but taking care of any WinCrypt BCrypt, whatever issues).

Is not rand_s VC++ specific ? There really are other compilers on Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

...

In the boostorg/uuid library there is a seed_rng class which is doing

...
similar things (however providing more entropy) and the library is header-only.

The rand_s function uses the operating system to generate cryptographically secure random numbers. It doesn't get better than that, maybe a call to RdRand https://en.wikipedia.org/wiki/RdRand#RDSEED on a current Intel CPU, which would use thermal noise (but you'll have to take Intel's word for it and just cross your fingers they didn't sell you out to a 3-letter agency).

degski

Gavin Lambert

11:28 p.m.

On 14/09/2017 10:21, Edward Diener wrote:

...

On 9/12/2017 9:27 AM, degski wrote:

Is not rand_s VC++ specific ? There really are other compilers on Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

rand_s is MSVCRT specific. But AFAIK gcc and clang both use MSVCRT on Windows anyway, so it should work (though might depend on the age of the CRT used). If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested, since this is the OS-level API.

James E. King, III

14 Sep 14 Sep

12:32 a.m.

RtlGenRandom is no supported on non-desktop UWP, however I'd like to thank everyone for their responses on this so far. I'm planning on submitting some changes to winapi and random, and will look at the existing random_device as a baseline given feedback on seed_rng, and work towards making the random library header-only. I'll be adding some comprehensive CI integration for travis and appveyor to make sure all the compile-time branches are being exercised. On Wed, Sep 13, 2017 at 7:28 PM, Gavin Lambert via Boost < boost@lists.boost.org> wrote:

...

On 14/09/2017 10:21, Edward Diener wrote:

...
On 9/12/2017 9:27 AM, degski wrote:

Is not rand_s VC++ specific ? There really are other compilers on Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

rand_s is MSVCRT specific. But AFAIK gcc and clang both use MSVCRT on Windows anyway, so it should work (though might depend on the age of the CRT used).

If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested, since this is the OS-level API.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman /listinfo.cgi/boost

degski

4:37 a.m.

On 14 September 2017 at 03:32, James E. King, III via Boost < boost@lists.boost.org> wrote:

...

RtlGenRandom is no supported on non-desktop UWP ...

This page https://docs.microsoft.com/en-us/windows/uwp/get-started/whats-a-uwp states that: "UWP apps use the Windows Runtime, a native API built into the operating system." ... and ... Some options for writing apps in UWP include: ... C++ backend ... So it should be possible to write: extern "C" int rand_s ( unsigned int* ); which tells the compiler not to worry about the existence of this function, "we'll give it to you at link-time". This works (and only works AFAIK), because it's a "C" function, a function without C++ name mangling. degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

Andrey Semashev

4:34 p.m.

On 09/14/17 03:32, James E. King, III via Boost wrote:

...

RtlGenRandom is no supported on non-desktop UWP, however I'd like to thank everyone for their responses on this so far. I'm planning on submitting some changes to winapi and random, and will look at the existing random_device as a baseline given feedback on seed_rng, and work towards making the random library header-only. I'll be adding some comprehensive CI integration for travis and appveyor to make sure all the compile-time branches are being exercised.

James, please, don't top post. http://www.boost.org/community/policy.html#quoting

...

On Wed, Sep 13, 2017 at 7:28 PM, Gavin Lambert via Boost < boost@lists.boost.org> wrote:

...
On 14/09/2017 10:21, Edward Diener wrote:

...
On 9/12/2017 9:27 AM, degski wrote:

Is not rand_s VC++ specific ? There really are other compilers on Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

rand_s is MSVCRT specific. But AFAIK gcc and clang both use MSVCRT on Windows anyway, so it should work (though might depend on the age of the CRT used).

If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested, since this is the OS-level API.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman /listinfo.cgi/boost

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

degski

4:24 a.m.

On 14 September 2017 at 02:28, Gavin Lambert via Boost < boost@lists.boost.org> wrote:

...

If in doubt, you could use RtlGenRandom instead, as Peter Dimov suggested, since this is the OS-level API.

rand_s calls RtlGenRandom (or any future API in case that might disappear. degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

degski

4:22 a.m.

On 14 September 2017 at 01:21, Edward Diener via Boost < boost@lists.boost.org> wrote:

...

Is not rand_s VC++ specific ? There really are other compilers on Windows other than VC++ ( I can test clang, gcc, and Intel for Windows).

It's Windows-specific, and has been part of the CRT since Windows XP. degski -- "*Ihre sogenannte Religion wirkt bloß wie ein Opiat reizend, betäubend, Schmerzen aus Schwäche stillend.*" - Novalis 1798

Andrey Semashev

12 Sep 12 Sep

2:39 p.m.

On 09/12/17 15:40, James E. King, III via Boost wrote:

...

Hi folks,

It looks like the only reason boostorg/random is not a header-only library is due to random_device (it is the only src/ file: https://github.com/boostorg/random/tree/develop/src). The older WinCrypt calls are not available in some Windows platforms going forward, so this is going to become a problem pretty quickly for the community.

In the boostorg/uuid library there is a seed_rng class which is doing similar things (however providing more entropy) and the library is header-only. In addition there are two filed defects - one in Trac and one in github issues, about uuid being incompatible with UWP applications (that's Windows universal apps). In uuid, I started adding support for BCrypt which is a Windows API supported on later versions of Windows, and once I changed my build flags to target Windows 10 and WINAPI_FAMILY_APP, I started getting build failures in random_device.

On further inspection it looks like the random library doesn't have support for BCrypt yet, and even worse as a source library one would need to build two variants - one for WinCrypt and one for BCrypt to support older and newer windows platforms. There are also some implications on MinGW, as the older original MinGW has support for WinCrypt and MinGW-64 supports both.

I'm going to propose that the implementation from boost::uuids::detail::seed_rng be moved into boost::random as the new random_device implementation. At the same time, boost::random will become header-only, and thus support setting _WIN32_WINNT at build time of consuming applications properly, and therefore support new Windows platforms properly.

At the same time I would like to change seed_rng such that it no longer silently ignores some return values, and does not disable any compiler warnings (deprecations, currently).

Thoughts?

I think that making Boost.Random header-only would be good. In addition I would suggest adding support for getrandom() on Linux and getentropy() on OpenBSD, as they are protected from file descriptor exhaustion. This can be done in a separate PR though. You will likely need to add BCrypt API to Boost.WinAPI - please, make a PR to it as well.

2630

Age (days ago)

2632

Last active (days ago)

List overview

Download

15 comments

6 participants

participants (6)

Andrey Semashev
degski
Edward Diener
Gavin Lambert
James E. King, III
Peter Dimov