MPL: Possible bug related to for_each and numbered vectors
Hi all, I stumbled across something very strange related to boost::mpl::for_each and numbered vectors and sets (as opposed to the variadic forms). The program below contains a typelist with 21 (i.e. more than BOOST_MPL_LIMIT_SET_SIZE) classes. This typelist is converted to a typelist containing pointers to those classes. Finally the main program calls for_each with that typelist to print the names of the types. This works without problems on Linux and Windows. But now: If you remove the include "boost/mpl/vector/vector30.hpp" everything still compiles fine without warnings under both operating systems. On Linux everything continues to work, whereas under Windows nothing is printed anymore. for_each does not loop through the typelist for an unknown reason. Everything works again, when reducing the number of classes to 20 (and adjusting the include to set20.hpp). From my understanding BOOST_MPL_LIMIT_SET_SIZE and its brothers and sisters should not have any impact on numbered sequences, only on variadic ones. But still it looks as if something very strange is happening here. Any comments are welcome, Martin #include <iostream> #include "boost/mpl/vector/vector30.hpp" #include "boost/mpl/set/set30.hpp" #include "boost/mpl/for_each.hpp" #include "boost/mpl/transform.hpp" #include "boost/mpl/inserter.hpp" #include "boost/mpl/vector.hpp" #include "boost/type_traits/add_pointer.hpp" class C1; class C2; class C3; class C4; class C5; class C6; class C7; class C8; class C9; class C10; class C11; class C12; class C13; class C14; class C15; class C16; class C17; class C18; class C19; class C20; class C21; typedef boost::mpl::set21<C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21 > ElementClasses; typedef boost::mpl::transform<ElementClasses,boost::add_pointer<boost::mpl::_1>, boost::mpl::inserter<boost::mpl::vector0<>, boost::mpl::push_back<boost::mpl::_1, boost::mpl::_2> > >::type ElementClassesAsPointer; class Test { public: Test(void) { boost::mpl::for_each<ElementClassesAsPointer> (*this); } template<class T> void operator() (T x) { std::cout << "Operator() called for type " << typeid(T).name() << std::endl; } }; int main(void) { Test test; } ____________ Virus checked by G DATA AntiVirus Version: AVKA 18.302 from 18.06.2008 Virus news: www.antiviruslab.com
AMDG Martin Apel wrote:
I stumbled across something very strange related to boost::mpl::for_each and numbered vectors and sets (as opposed to the variadic forms). The program below contains a typelist with 21 (i.e. more than BOOST_MPL_LIMIT_SET_SIZE) classes. This typelist is converted to a typelist containing pointers to those classes. Finally the main program calls for_each with that typelist to print the names of the types. This works without problems on Linux and Windows. But now: If you remove the include "boost/mpl/vector/vector30.hpp" everything still compiles fine without warnings under both operating systems. On Linux everything continues to work, whereas under Windows nothing is printed anymore.
I don't think that it should compile on windows. I assume that you are using msvc on Windows and gcc on Linux?
for_each does not loop through the typelist for an unknown reason. Everything works again, when reducing the number of classes to 20 (and adjusting the include to set20.hpp). From my understanding BOOST_MPL_LIMIT_SET_SIZE and its brothers and sisters should not have any impact on numbered sequences, only on variadic ones. But still it looks as if something very strange is happening here.
Here's the result of the transform is on msvc 9.0 boost::mpl::vector21<C21 *,C20 *,C19 *,C18 *,C17 *,C16 *,C15 *,C14 *,C13 *,C12 *,C11 *,C10 *,C9 *,C8 *,C7 *,C6 *,C5 *,C4 *,C3 *,C2 *,C1 *> Without <boost/mpl/vector/vector30.hpp> the result is boost::mpl::push_back<boost::mpl::vector20<C21 *,C20 *,C19 *,C18 *,C17 *,C16 *,C15 *,C14 *,C13 *,C12 *,C11 *,C10 *,C9 *,C8 *,C7 *,C6 *,C5 *,C4 *,C3 *,C2 *>,C1 *> The reason? On compilers without typeof, including msvc, push_back on a vector0 gives a vector1, push_back on a vector1 gives a vector2, and so on. The unnumbered form is specialized for every possible number of arguments and inherits from the appropriate numbered form. Now for the strange part--when an mpl lambda expression is evaluated it checks to see whether the result of substituting placeholders is a metafunction, in other words it looks for a nested typedef ::type. If this is present, then it will return that typedef, otherwise, it will return type type unchanged. For example the expressions boost::mpl::apply<std::vector<_1>, int>::type evaluates to std::vector<int>. Now, if you don't #include <boost/mpl/vector/vector30.hpp>, there will be no specialization of push_back for vector20, thus mpl::lambda will think that the push_back is not a metafunction and return it unchanged. Finally, when mpl::begin and mpl::end are called on a type that is not an MPL sequence, they return mpl::void_. Since begin and end return the same type, for_each decides that it is dealing with an empty sequence. Yuck. In Christ, Steven Watanabe
Hi all, Steven Watanabe wrote:
AMDG
Martin Apel wrote:
I stumbled across something very strange related to boost::mpl::for_each and numbered vectors and sets (as opposed to the variadic forms). The program below contains a typelist with 21 (i.e. more than BOOST_MPL_LIMIT_SET_SIZE) classes. This typelist is converted to a typelist containing pointers to those classes. Finally the main program calls for_each with that typelist to print the names of the types. This works without problems on Linux and Windows. But now: If you remove the include "boost/mpl/vector/vector30.hpp" everything still compiles fine without warnings under both operating systems. On Linux everything continues to work, whereas under Windows nothing is printed anymore.
I don't think that it should compile on windows. I assume that you are using msvc on Windows and gcc on Linux?
Yes, exactly.
for_each does not loop through the typelist for an unknown reason. Everything works again, when reducing the number of classes to 20 (and adjusting the include to set20.hpp). From my understanding BOOST_MPL_LIMIT_SET_SIZE and its brothers and sisters should not have any impact on numbered sequences, only on variadic ones. But still it looks as if something very strange is happening here.
Here's the result of the transform is on msvc 9.0 boost::mpl::vector21<C21 *,C20 *,C19 *,C18 *,C17 *,C16 *,C15 *,C14 *,C13 *,C12 *,C11 *,C10 *,C9 *,C8 *,C7 *,C6 *,C5 *,C4 *,C3 *,C2 *,C1 *>
Without <boost/mpl/vector/vector30.hpp> the result is boost::mpl::push_back<boost::mpl::vector20<C21 *,C20 *,C19 *,C18 *,C17 *,C16 *,C15 *,C14 *,C13 *,C12 *,C11 *,C10 *,C9 *,C8 *,C7 *,C6 *,C5 *,C4 *,C3 *,C2 *>,C1 *>
The reason? On compilers without typeof, including msvc, push_back on a vector0 gives a vector1, push_back on a vector1 gives a vector2, and so on. The unnumbered form is specialized for every possible number of arguments and inherits from the appropriate numbered form.
Now for the strange part--when an mpl lambda expression is evaluated it checks to see whether the result of substituting placeholders is a metafunction, in other words it looks for a nested typedef ::type. If this is present, then it will return that typedef, otherwise, it will return type type unchanged. For example the expressions boost::mpl::apply<std::vector<_1>, int>::type evaluates to std::vector<int>. Now, if you don't #include <boost/mpl/vector/vector30.hpp>, there will be no specialization of push_back for vector20, thus mpl::lambda will think that the push_back is not a metafunction and return it unchanged.
Finally, when mpl::begin and mpl::end are called on a type that is not an MPL sequence, they return mpl::void_. Since begin and end return the same type, for_each decides that it is dealing with an empty sequence.
Hm, typeof is not yet part of the C++ standard, as far as I know. This means, that Boost MPL relies on non-standard behaviour to work correctly (in this case). What makes this very problematic is, that there is no error or warning, simply different behaviour across platforms. This means I will have to stumble through each case, where the availability of the typeof operator makes a difference. To me it looks, as if apply should not compile, if its argument is not a metafunction. But I don't have enough overview of the internal workings of MPL to estimate the consequences of this. Or is there any way to at least notice, that something is going wrong without testing the software relying on MPL completely? Regards, Martin ____________ Virus checked by G DATA AntiVirus Version: AVKB 18.318 from 23.06.2008
Hi Martin, (I assume you missed my reply to your original post -- http://article.gmane.org/gmane.comp.lib.boost.user/36892)
Hm, typeof is not yet part of the C++ standard, as far as I know. This means, that Boost MPL relies on non-standard behaviour to work correctly (in this case). What makes this very problematic is, that there is no error or warning, simply different behaviour across platforms.
It would be very problematic *if* the behaviour you witnessed was somehow inherent / by design. However, it's not. It was simply a bug, albeit a nasty one, which I fixed in the trunk -- http://svn.boost.org/trac/boost/changeset/46546
This means I will have to stumble through each case, where the availability of the typeof operator makes a difference.
Please see my post above. In short, the referenced changelist gives us a reasonable degree of assurance that you don't have to do that. The original bug that would lead to a silent failure dependening of the typeof availability had a limited scope (push_back/push_front) and has been fixed, and the related issue of sequence algorithms/metafunctions silently doing nothing if passed a non-sequence (regardless on typeof) has been also addressed (although not completely). We also will be working on adding more test coverage for this particular aspect of the library (http://svn.boost.org/trac/boost/ticket/2041).
To me it looks, as if apply should not compile, if its argument is not a metafunction.
While the 'apply' heuristic is increasing the possibility of your code silently doing something else from what you think it's doing, it behaves consistently across all platforms, and it was not the culprit in your original example.
But I don't have enough overview of the internal workings of MPL to estimate the consequences of this. Or is there any way to at least notice, that something is going wrong without testing the software relying on MPL completely?
Please let us know if the above addresses your concerns. -- Aleksey Gurtovoy MetaCommunications Engineering
Hi Martin,
I stumbled across something very strange related to boost::mpl::for_each and numbered vectors and sets (as opposed to the variadic forms). The program below contains a typelist with 21 (i.e. more than BOOST_MPL_LIMIT_SET_SIZE) classes. This typelist is converted to a typelist containing pointers to those classes. Finally the main program calls for_each with that typelist to print the names of the types. This works without problems on Linux and Windows. But now: If you remove the include "boost/mpl/vector/vector30.hpp" everything still compiles fine without warnings under both operating systems. On Linux everything continues to work, whereas under Windows nothing is printed anymore.
It's a bug in the library's diagnostics (or, rather, a lack of such) -- please see below.
for_each does not loop through the typelist for an unknown reason. Everything works again, when reducing the number of classes to 20 (and adjusting the include to set20.hpp). From my understanding BOOST_MPL_LIMIT_SET_SIZE and its brothers and sisters should not have any impact on numbered sequences, only on variadic ones.
That's correct.
But still it looks as if something very strange is happening here.
Indeed. There are several factors at play here: 1. To be able to 'push_back' into a 'vectorN' on a compiler without 'typeof' support you have to have a 'vectorN+1' definition included. If you don't have it included, you will get a compilation error: #include "boost/mpl/vector/vector10_c.hpp" #include "boost/mpl/push_back.hpp" using namespace boost::mpl; typedef push_back< vector10_c<int,1,2,3,4,5,6,7,8,9,10> , int_<11> >::type t; > test.cpp(9) : error C2039: 'type' : is not a member of 'boost::mpl::push_back<Sequence,T>' > with > [ > Sequence=boost::mpl::vector10_c<int,1,2,3,4,5,6,7,8,9,10>, > T=boost::mpl::int_<11> > ] ... except when you don't, like you experienced first hand. 2. The reason your code doesn't result in the error above is that it doesn't invoke 'push_back' directly -- it supplies it as an output operation to the inserter. When the inserter does eventually invoke 'push_back' on 'vector20<...>' (at the last transformation step), the invocation is done through the 'apply' metafunction and is basically equivalent to this: typedef apply< push_back<_1,_2>, vector20<...>, C21 >::type t; This, of course, shouldn't make any difference and should produce the same error, but it doesn't. Instead, it results in 't' being an internal implementation type which has nothing to do with vector (nor any other sequence). That's where the bug (the absence of proper diagnostics) is. 3. Due to the lack of diagnostics in the previous step, 'ElementClassesAsPointer' ends up being an typedef to an internal type that is not a sequence. On a surface, it seems that passing a non-sequence to 'for_each' should still result in a compilation error. In fact, 'for_each' and other algorithms/metafunctions in the library almost never explicitly check conformace of the provided template parameters to their corresponding concepts. In absence of the explicit concept comformance verification, invocation of a sequence algorithm on a non-sequence type is almost guaranteed to be a no-op because of the following piece of the 'begin'/'end' specification (http://www.boost.org/doc/libs/1_35_0/libs/mpl/doc/refmanual/begin.html): [..] If the argument is not a Forward Sequence, returns void_. Thus the observed effect of boost::mpl::for_each<ElementClassesAsPointer> (*this); in your example. The absence of concept checks can be argued to be another bug. I've just checked in a fix for the bug in step #2, and also a number of related fixes that together greatly reduce the chance of silent diagnoctic failures in other similar situations -- http://svn.boost.org/trac/boost/changeset/46546. If you are not compiling against the trunk, it should be safe to apply the diff to your local Boost version as well. The proper concept checks is something that will probably have to wait until C++0x is out in the field. Thank you for taking time to report this, -- Aleksey Gurtovoy MetaCommunications Engineering
participants (1)
-
Aleksey Gurtovoy -
Martin Apel -
Steven Watanabe