Good morning, We are developing a system which heavily uses memory mapped files. These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS. We are having some problems loading memory mapped files created under Unix into Android. Is this scenario even possible ? Are the memory mapped files compatible between systems ? Hope you will be able to shed some light on this. Best regards Simon
On 21/01/2019 22:41, Simon Giddings wrote:
We are developing a system which heavily uses memory mapped files. These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS. We are having some problems loading memory mapped files created under Unix into Android.
In what context? Using some Boost library or in general?
Is this scenario even possible ? Are the memory mapped files compatible between systems ?
They can be, but are not guaranteed to be, depending on what you're storing in the memory. Binary layout of class types is usually not guaranteed cross-platform (or cross-compiler, for that matter). Anything involving pointers usually won't work at all, even cross-process or cross-session, let alone cross-platform. You can also have issues with padding or value endian ordering, depending on the specific platforms involved.
We have already taken into account the problem of pointers and also endian ordering. When doing a hex dump of the files produced under unix and one produced under android, using the same data content, we have noticed that there are byte differences. So, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ? On 23/01/2019 01:46, Gavin Lambert via Boost-users wrote:
On 21/01/2019 22:41, Simon Giddings wrote:
We are developing a system which heavily uses memory mapped files. These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS. We are having some problems loading memory mapped files created under Unix into Android.
In what context? Using some Boost library or in general?
Is this scenario even possible ? Are the memory mapped files compatible between systems ?
They can be, but are not guaranteed to be, depending on what you're storing in the memory.
Binary layout of class types is usually not guaranteed cross-platform (or cross-compiler, for that matter).
Anything involving pointers usually won't work at all, even cross-process or cross-session, let alone cross-platform.
You can also have issues with padding or value endian ordering, depending on the specific platforms involved. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
On 1/23/19 12:19 AM, Simon Giddings via Boost-users wrote:
We have already taken into account the problem of pointers and also endian ordering. When doing a hex dump of the files produced under unix and one produced under android, using the same data content, we have noticed that there are byte differences. So, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ?
This is the purpose of the Boost Serialization library. I'm quite sure it (or likely any serialization library) would be quite suitable for for addressing this problem. Robert Ramey
We are developing a system which heavily uses memory mapped files. These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS. We are having some problems loading memory mapped files created under Unix into Android.
Is this scenario even possible ? Are the memory mapped files compatible between systems ?
This kind of question has nothing to do with Boost, and would be better asked on Stackoverflow. However yes Android supports memory mapped files just fine. Free address space is very limited though, so what works on 64 bit workstations may not work on 32 bit Android. Also, Intel CPUs are not ARM CPUs, padding of structs will be different, floating point numbers may have a different representation or be interpreted a bit differently, there are a few other differences. With discipline memory mapped files usable on both architectures can be devised, but usually with a performance penalty on all platforms. In the end there is no free lunch. Niall
Sorry, but this is a very specific Boost question as we are using the Boost memory mapping system. So, if I understand correctly, you are saying that Boost does / cannot garantee compatibility between OS and cpu architecture ? This would mean that Boost leverages heavily the underlying OS services where available. Having said that, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ? On 23/01/2019 02:33, Niall Douglas via Boost-users wrote:
We are developing a system which heavily uses memory mapped files. These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS. We are having some problems loading memory mapped files created under Unix into Android.
Is this scenario even possible ? Are the memory mapped files compatible between systems ? This kind of question has nothing to do with Boost, and would be better asked on Stackoverflow.
However yes Android supports memory mapped files just fine. Free address space is very limited though, so what works on 64 bit workstations may not work on 32 bit Android. Also, Intel CPUs are not ARM CPUs, padding of structs will be different, floating point numbers may have a different representation or be interpreted a bit differently, there are a few other differences.
With discipline memory mapped files usable on both architectures can be devised, but usually with a performance penalty on all platforms. In the end there is no free lunch.
Niall _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
On 1/23/19 3:15 AM, Simon Giddings via Boost-users wrote:
Sorry, but this is a very specific Boost question as we are using the Boost memory mapping system.
So, if I understand correctly, you are saying that Boost does / cannot garantee compatibility between OS and cpu architecture ? This would mean that Boost leverages heavily the underlying OS services where available. Having said that, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ?
This is very much a generic language question, not a Boost question. C++ make NO guarantee that in-memory organization of data structures will match between different architectures, and there is nothing Boost can do to make those structures match between architectures. You can 'serialize' data in a architecture agnostic manner, but in general, that will be nothing like a memory dump. To be totally general, you need to dump the values in something like an ASCII representation. If you can be sure that individual types have the same representation, you can output that as a string of bytes for the individual types, and reconstruct the types on reading. Do NOT expect class level object to be streamed out at the byte level and be restore-able at the byte level, so it will not be anything like a memory map file that can be just used as is. -- Richard Damon
C++ make NO guarantee that in-memory organization of data structures will match between different architectures, and there is nothing Boost can do to make those structures match between architectures.
It's even stronger than that: the C++ standard makes no guarantee that in-memory representation of storage does not change between program executions, let alone between different compilers or architectures. To use mapped memory at all outside a single program execution is pure undefined behaviour. You are literally on your own wrt the standard. (I have a proposal in the works for WG21 SG12 which will propose how to add support for shared and paged memory to C and C++, but even it specifically excludes the ability to use storage by programs other than the program which constructed the objects. And even that, the standard requiring that the same program binary always uses the same in-memory representation, will be very, very controversial e.g. imagine JITed or translated C++ for example) Niall
On 1/23/19 9:12 AM, Niall Douglas via Boost-users wrote:
C++ make NO guarantee that in-memory organization of data structures will match between different architectures, and there is nothing Boost can do to make those structures match between architectures. It's even stronger than that: the C++ standard makes no guarantee that in-memory representation of storage does not change between program executions, let alone between different compilers or architectures. To use mapped memory at all outside a single program execution is pure undefined behaviour. You are literally on your own wrt the standard.
(I have a proposal in the works for WG21 SG12 which will propose how to add support for shared and paged memory to C and C++, but even it specifically excludes the ability to use storage by programs other than the program which constructed the objects. And even that, the standard requiring that the same program binary always uses the same in-memory representation, will be very, very controversial e.g. imagine JITed or translated C++ for example)
Niall
While the standard doesn't explicitly define it, at least a POD which doesn't contain pointers generally should maintain representation within the same executable, even if run multiple times. The offsetof each of the members is defined to be a compile time constant, so unless the compiler uses some from of whole-program optimization to invoke the as-if rule for structure layout, it can't be a run-time decision. The representation of each of the fundamental members needs to be documented, so unless the implementation documents a possible change of representation at executable time, that will be consistent. At a higher level, platform ABIs will generally define the representation of structures, so that provides a stronger promise (and I suspect all the implementations that Boost supports define themselves to conforming to some platform ABI). This is needed for the linking together of independently compiled modules. Pointers will be an issue, as there is generally no assurance that the program will run is the same memory space run to run (except on many embedded platforms) so they would need to be fixed, and in general there is no standard way to do this. Non-PODs also have this issue, in part because they may include as part of their representation pointers to related information (like the v-table for virtual functions as an example). -- Richard Damon
It's even stronger than that: the C++ standard makes no guarantee that in-memory representation of storage does not change between program executions, let alone between different compilers or architectures. To use mapped memory at all outside a single program execution is pure undefined behaviour. You are literally on your own wrt the standard.
While the standard doesn't explicitly define it, at least a POD which doesn't contain pointers generally should maintain representation within the same executable, even if run multiple times.
It depends on your execution environment. An implementation based on an interpreter or JIT may readily perform WPO which changes compile time constants across program runs. You may not think such execution environments common, but I'd remind you of C++ on GPUs, or C++ cross compiled into Javascript. Further on down the line we may have Modules from a Modules store be late stage optimised into locally optimised binaries. Padding, in particular, might be optimisable, especially as we start to support ever larger integer types. Padding with 512-bit integers, for example, is wasteful. What I will agree with is that for a given version of a given toolchain on a given architecture e.g. GCC 7.3 on x64, you currently get stronger guarantees. But you shouldn't rely on anything not explicitly guaranteed by the standard or your toolchain vendor. Niall
On 23/01/2019 21:15, Simon Giddings wrote:
Sorry, but this is a very specific Boost question as we are using the Boost memory mapping system.
You still haven't indicated what library you're talking about. There is no "Boost memory mapping system"; there may or may not be a particular library you are using. But, as previously stated, C++ makes no particular guarantees about the memory layout of class objects and only very minimal guarantees about the layout of POD types. This is especially true cross-platform and cross-toolset, but can bite you when you update the compiler or runtimes within a single platform, or even potentially from one run to another (eg. hash/map ordering can change). If you want portability, then you need to use a serialisation library which implements portability (not all do) along with compatible data structures; but this is an entirely different animal from plain memory mapping.
participants (1)
-
Gavin Lambert -
Niall Douglas -
Richard Damon -
Robert Ramey -
Simon Giddings