[serialization] char[] and double
We have thousands of 'packed' 'C' structs (2100+) shared between C and C++ modules & apps. These structures do NOT contain any pointers, all data is contiguous so that it may be transferred directly from one machine's memory to another machine's memory without translation (32 bit Windows on Intel), e.g. #ifdef __cplusplus extern "C" { #endif typedef struct NODE_MGR { char ApplName[8]; char CompName[8]; } NODE_MGR; typedef struct ROUTE_CONTROL { UCHAR msg_type; ULONG req_num; char src_domain_id[ 15 ]; char src_machine_id[ 8 ]; char src_proc_id[ 32 ]; char dest_domain_id[ 15 ]; char dest_machine_id[ 8 ]; char dest_proc_id[ 32 ]; NODE_MGR PrimeNodeMgr; NODE_MGR BkupNodeMgr; UINT parmlen; UINT datalen; /* data[] is actually 'parmlen + datalen' bytes * in length (64K max) and contains arrays of * other struct's who's types vary based on the values * of 'msg_type' and 'req_num'. */ char data[ 1 ]; } ROUTE_CONTROL; /* hundreds of other struct defs here... */ #ifdef __cplusplus } #endif Currently, these structs are written directly from the workstation's memory to a TCP/IP socket connected to a remote server process which reads them from the socket directly into memory for immediate use by the server program (i.e. 'packed' binary structs are transfered over the net from one 32 bit Windows machine to another 32 bit Windows machine). In the near future the workstations will continue to be 32 bit Windows, but the servers may be 32 bit Linux on Intel, or 64 bit Linux on AMD, or 64 bit Linux on PowerPC, or 64 bit Solaris on Sparc. This means that we can no longer send/receive 32 bit Intel 'packed' binary structs directly from the workstation to the server. I had hoped to use Boost Archives to serialize the structs to a std::string (via std::strstream), send it across the net, then deserialize the structs at the server. However, the structs have thousands of members of the form 'char[N]' representing char arrays which are NOT nul-terminated. I can not get these struct members to serialize; I get compile errors, e.g. this fails to compile: ar & fld; // where fld is: char[200] What's the secret trick? The structs also contain many 'double' fields. If the precision is the (default?) 6 when serialized, that won't be enough. Is there a way to control the precision of doubles when they are written to a text archive? Thanks, Larry
Larry Smith wrote:
ar & fld; // where fld is: char[200]
What's the secret trick?
There a couple of ways. On is to derive your archive from text_?archive and add special handling for arrays. Look at demo_fast_archive which does this for arrays - (It uses binary_?archive as a base - but that's not important here).
The structs also contain many 'double' fields. If the precision is the (default?) 6 when serialized, that won't be enough. Is there a way to control the precision of doubles when they are written to a text archive?
Hmmm - the precision saved in a text archive is determined by the numeric traits on the machine creating the archive so I would hope that the default precision would be the exact correct one. Also it might depend on the stream implementation. I don't think fact that a particular value is rendered to 6 places means anything. That is I would expect that the value 2.0 is rendered as "2.0" rather than "2.000000000000000" . I don't remember off hand where the precision is set. I would expect its easy to change in the stream after the archive is opened. Robert Ramey
Thanks, Larry
Larry Smith wrote: I can not get these struct members to
serialize; I get compile errors, e.g. this fails to compile:
ar & fld; // where fld is: char[200]
Hmmm - works for me. The following example compiles just fine. #include <sstream> #include <boost/archive/text_oarchive.hpp> struct x { char y[200]; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & y; } }; int main(int argc, char *argv[]){ std::stringstream os; boost::archive::text_oarchive oa(os); const x x1; oa & x1; } Robert Ramey
Robert Ramey wrote:
Larry Smith wrote:
I can not get these struct members to
serialize; I get compile errors, e.g. this fails to compile:
ar & fld; // where fld is: char[200]
Hmmm - works for me. The following example compiles just fine.
#include <sstream>
#include <boost/archive/text_oarchive.hpp>
struct x {
char y[200];
template<class Archive>
void serialize(Archive & ar, const unsigned int version){
ar & y;
}
};
int main(int argc, char *argv[]){
std::stringstream os;
boost::archive::text_oarchive oa(os);
const x x1;
oa & x1;
}
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
The structs we have to deal with are 'C' structs, not C++. They are included in approx 9000 source files; 90% of those source files are 'C', only 10% are C++. All of the source is compiled with 'pack 1' ('/Zp1 on Windows, __attribute__ ((packed)) on Linux gcc/g++) to eliminate 'pad fields' within the structs; all of the source files depend on this packed layout, so it can not be changed - things would break. Once we move to non-Intel servers, 'packed' will no longer be used on the server side; this is another reason to serialize-xfer-deserialize the data - the struct layouts/sizes will be very different between the workstation and server. Here's a highly reduced example: ----------------------- /* cmdef.h */ /* NOTE: on Linux/Intel with gcc/g++ 'ADPACKED' is * defined as '__attribute__ ((packed))'. * on Windows the '/Zp1' compile option is * used to pack the structs and 'ADPACKED' * is defined as empty. */ #ifdef __cplusplus extern "C" { #endif typedef struct ADPACKED NODE_MGR { char ApplName[8]; char CompName[8]; } NODE_MGR; #ifdef __cplusplus } #endif --------------------- /* arch.hpp */ #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include "cmdef.h" namespace boost { namespace serialization { template<class Archive> void serialize(Archive & ar, struct NODE_MGR & st, const unsigned int version) { ar & st.ApplName; // line 19 in err msg ar & st.CompName; // line 20 in err msg } } // end namespace serialization */ } // end of namespace boost */ -------------------- /* arch.cpp */ #include <sstream> #include "arch.hpp" int main() { struct NODE_MGR nm; std::ostringstream ofs; boost::archive::text_oarchive oa(ofs); oa << nm; // line 15 in err msg return 0; } ------------------- Compiling arch.cpp produces these errors: arch.hpp:19: error: cannot bind packed field ‘st->NODE_MGR::ApplName’ to ‘char (&)[8] arch.hpp:20: error: cannot bind packed field ‘st->NODE_MGR::CompName’ to ‘char (&)[8] Removing the 'packed' attribute produces a different set of errors: /usr/include/boost/archive/detail/oserializer.hpp: In function ‘void boost::archive::save(Archive&, T&) [with Archive = boost::archive::text_oarchive, T = NODE_MGR]’: /usr/include/boost/archive/basic_text_oarchive.hpp:78: instantiated from ‘void boost::archive::basic_text_oarchive<Archive>::save_override(T&, int) [with T = NODE_MGR, Archive = boost::archive::text_oarchive]’ /usr/include/boost/archive/detail/interface_oarchive.hpp:85: instantiated from ‘Archive& boost::archive::detail::interface_oarchive<Archive>::operator<<(T&) [with T = NODE_MGR, Archive = boost::archive::text_oarchive]’ arch.cpp:15: instantiated from here /usr/include/boost/archive/detail/oserializer.hpp:566: error: invalid application of ‘sizeof’ to incomplete type ‘boost::STATIC_ASSERTION_FAILURE<false>’ However, on the workstation side the structs MUST remain 'packed'; otherwise thousands of source files will have to be modified. Is there any way to make this work with boost archives, or should I explore other options? Thanks, Larry
#include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp>
#include "cmdef.h"
namespace boost { namespace serialization {
template<class Archive> void serialize(Archive & ar, struct NODE_MGR & st, const unsigned int version) { ar & st.ApplName; // line 19 in err msg ar & st.CompName; // line 20 in err msg }
} // end namespace serialization */ } // end of namespace boost */
--------------------
/* arch.cpp */
#include <sstream>
#include "arch.hpp"
int main() { const struct NODE_MGR nm; // this will fix errors in second case - see rationale std::ostringstream ofs; boost::archive::text_oarchive oa(ofs);
oa << nm; // line 15 in err msg
return 0; }
Compiling arch.cpp produces these errors:
arch.hpp:19: error: cannot bind packed field st->NODE_MGR::ApplName to char (&)[8]
arch.hpp:20: error: cannot bind packed field st->NODE_MGR::CompName to char (&)[8]
However, on the workstation side the structs MUST remain 'packed'; otherwise thousands of source files will have to be modified.
Is there any way to make this work with boost archives, or should I explore other options?
I don't see how this could be addressed from within the serialization library. It looks like anthing that looks like ostream os; os << nm; is going to fail as well. I suspect that your strategy is going to be foiled whether you use the serialization or some other method. Robert Ramey
Robert Ramey wrote:
#include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp>
#include "cmdef.h"
namespace boost { namespace serialization {
template<class Archive> void serialize(Archive & ar, struct NODE_MGR & st, const unsigned int version) { ar & st.ApplName; // line 19 in err msg ar & st.CompName; // line 20 in err msg }
} // end namespace serialization */ } // end of namespace boost */
--------------------
/* arch.cpp */
#include <sstream>
#include "arch.hpp"
int main() { const struct NODE_MGR nm; // this will fix errors in second case - see rationale std::ostringstream ofs; boost::archive::text_oarchive oa(ofs);
oa << nm; // line 15 in err msg
return 0; }
Compiling arch.cpp produces these errors:
arch.hpp:19: error: cannot bind packed field ‘st->NODE_MGR::ApplName’ to ‘char (&)[8]
arch.hpp:20: error: cannot bind packed field ‘st->NODE_MGR::CompName’ to ‘char (&)[8]
However, on the workstation side the structs MUST remain 'packed'; otherwise thousands of source files will have to be modified.
Is there any way to make this work with boost archives, or should I explore other options?
I don't see how this could be addressed from within the serialization library. It looks like anthing that looks like
ostream os; os << nm;
is going to fail as well. I suspect that your strategy is going to be foiled whether you use the serialization or some other method.
Robert Ramey
------------------------------------------------------------------------
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Ok, I see how 'const struct NODE_MGR nm' fixes part of the problem... I don't understand why the lib should care about the data alignment of the struct's (packed versus unpacked). If I remove 'packed' from the struct's it builds & works OK; with 'packed' it won't compile. On the server side (the receiver) 'packed' won't be used, but on the workstation side (the sender) 'packed' must be used to keep from breaking 100+ existing workstation apps. Thanks for the help. Larry
Larry Smith wrote:
I don't understand why the lib should care about the data alignment of the struct's (packed versus unpacked).
The library doesn't; your compiler does, and it's wrong in this case. char[8] requires no padding or alignment, so it should bind to char(&)[8] in either mode. You might be able to use a reinterpret_cast to silence the compiler.
Peter Dimov wrote:
Larry Smith wrote:
I don't understand why the lib should care about the data alignment of the struct's (packed versus unpacked).
The library doesn't; your compiler does, and it's wrong in this case. char[8] requires no padding or alignment, so it should bind to char(&)[8] in either mode. You might be able to use a reinterpret_cast to silence the compiler.
Yes, the problem is that the real struct's have int's, long's, double's, and char[] members intermixed. The existing Windows code requires that the struct's remain packed (it's a long story with roots in the original implemenation 15 years ago). Thanks, Larry
participants (3)
-
Larry I Smith -
Larry Smith
-
Peter Dimov -
Robert Ramey