Max Boost Fusion tuple parameters

newer
[coroutine] Resuming of coroutine...

Rodrigo Madera

10 Jun 2013 10 Jun '13

6:40 p.m.

Hello, What is the maximum Boost Fusion tuple param count? I was thinking of using it for about 200-300 elements. Anyone done this? I've heard of defines for this, but never actually *knew* of anyone actually doing it. Thanks, Rodrigo Madera

Attachments:

attachment.html (text/html — 1.1 KB)

Show replies by date

Larry Evans

11 Jun 11 Jun

2:02 p.m.

On 06/10/13 13:40, Rodrigo Madera wrote:

...

Hello,

What is the maximum Boost Fusion tuple param count?

I was thinking of using it for about 200-300 elements.

Anyone done this?

I've heard of defines for this, but never actually *knew* of anyone actually doing it.

Thanks, Rodrigo Madera

I found some relevant information on this by: 1) going to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/index.html 2) then following the _Class template tuple_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/tuple/class... 3) then following the _vector_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/container/v... 4) which said: You may define the preprocessor constant FUSION_MAX_VECTOR_SIZE before including any Fusion header to change the default. Example: #define FUSION_MAX_VECTOR_SIZE 20 HTH. -regards, Larry

Igor R

2:31 p.m.

...

...
What is the maximum Boost Fusion tuple param count?

I was thinking of using it for about 200-300 elements.

Anyone done this?

I've heard of defines for this, but never actually *knew* of anyone actually doing it.

Thanks, Rodrigo Madera

I found some relevant information on this by:

1) going to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/index.html 2) then following the _Class template tuple_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/tuple/class... 3) then following the _vector_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/container/v...

4) which said: You may define the preprocessor constant FUSION_MAX_VECTOR_SIZE before including any Fusion header to change the default. Example:

#define FUSION_MAX_VECTOR_SIZE 20

Template argument number in a template declaration is limited by compilers. For example, in MSVC10/11 it's limited to 64. So I doubt setting FUSION_MAX_VECTOR_SIZE to 200 would work.

Larry Evans

3:11 p.m.

On 06/11/13 09:31, Igor R wrote:

...

...
...
What is the maximum Boost Fusion tuple param count?

I was thinking of using it for about 200-300 elements.

Anyone done this?

I've heard of defines for this, but never actually *knew* of anyone actually doing it.

Thanks, Rodrigo Madera

I found some relevant information on this by:

1) going to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/index.html 2) then following the _Class template tuple_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/tuple/class... 3) then following the _vector_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/container/v...

4) which said: You may define the preprocessor constant FUSION_MAX_VECTOR_SIZE before including any Fusion header to change the default. Example:

#define FUSION_MAX_VECTOR_SIZE 20

Template argument number in a template declaration is limited by compilers. For example, in MSVC10/11 it's limited to 64. So I doubt setting FUSION_MAX_VECTOR_SIZE to 200 would work.

Thanks Igor. I had no idea about that :( In this case, I'd guess the only solution for Rodrigo is to use Christopher Schmidt's variadic tuples: http://article.gmane.org/gmane.comp.lib.boost.devel/194322 I remember (a year or more ago) looking at the Christopher's implementation and, IIRC, it used a combination of the boost preprocessor library and variadic templates to speed the compilation times. Again, IIRC, Christoper built up a N*CHUNK size tuple of T element by creating N structs each containing CHUNK elements of type T. I think the N structs were recursively inherited; so that struct tuple_chunked<N> inherited from tuple_chunked<N-1>. Thus, if CHUNK were 64 (in the case of MSVC`10/11, then creating a tuple of 200 would just require about 200/64 template instantiations. The last I heard about Christopher's effort was this post: http://article.gmane.org/gmane.comp.lib.boost.devel/234501 The code location (I don't know if it's still valid) was give in post: http://article.gmane.org/gmane.comp.lib.boost.devel/228820 HTH. -regards, Larry

Larry Evans

6:52 p.m.

On 06/11/13 10:11, Larry Evans wrote:

...

On 06/11/13 09:31, Igor R wrote:

...
...
...
What is the maximum Boost Fusion tuple param count?

I was thinking of using it for about 200-300 elements.

Anyone done this?

I've heard of defines for this, but never actually *knew* of anyone actually doing it.

Thanks, Rodrigo Madera

I found some relevant information on this by:

1) going to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/index.html 2) then following the _Class template tuple_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/tuple/class... 3) then following the _vector_ link to: http://www.boost.org/doc/libs/1_53_0/libs/fusion/doc/html/fusion/container/v...

4) which said: You may define the preprocessor constant FUSION_MAX_VECTOR_SIZE before including any Fusion header to change the default. Example:

#define FUSION_MAX_VECTOR_SIZE 20

Template argument number in a template declaration is limited by compilers. For example, in MSVC10/11 it's limited to 64. So I doubt setting FUSION_MAX_VECTOR_SIZE to 200 would work.

Thanks Igor. I had no idea about that :(

In this case, I'd guess the only solution for Rodrigo is to use Christopher Schmidt's variadic tuples:

http://article.gmane.org/gmane.comp.lib.boost.devel/194322

I remember (a year or more ago) looking at the Christopher's implementation and, IIRC, it used a combination of the boost preprocessor library and variadic templates to speed the compilation times. Again, IIRC, Christoper built up a N*CHUNK size tuple of T element by creating N structs each containing CHUNK elements of type T. I think the N structs were recursively inherited; so that struct tuple_chunked<N> inherited from tuple_chunked<N-1>. Thus, if CHUNK were 64 (in the case of MSVC`10/11, then creating a tuple of 200 would just require about 200/64 template instantiations.

The last I heard about Christopher's effort was this post:

http://article.gmane.org/gmane.comp.lib.boost.devel/234501

The code location (I don't know if it's still valid) was give in post:

http://article.gmane.org/gmane.comp.lib.boost.devel/228820

I probably jumped to the conclusion that "template argument number" meant "number of template instantiations". Sorry for that mistake. I guess "template argument number" means the maximum arity of any template. IOW, if "template argument number" was 5, then any template taking over 5 arguments would cause the MSVC10/11 compiler to diagnose an error. IOW: tuple would be OK, but: tuple would cause MSV10/11 to issue a compiler error. Is that right? Also, I also read somewhere that MSVC had not yet implemented variadic templates: http://stackoverflow.com/questions/9589192/how-do-i-change-the-number-of-tem... Sorry for noise. -regards, Larry

Igor R

12 Jun 12 Jun

9:54 a.m.

...

I probably jumped to the conclusion that "template argument number" meant "number of template instantiations". Sorry for that mistake. I guess "template argument number" means the maximum arity of any template. IOW, if "template argument number" was 5, then any template taking over 5 arguments would cause the MSVC10/11 compiler to diagnose an error. IOW:

tuple

would be OK, but:

tuple

would cause MSV10/11 to issue a compiler error. Is that right?

Yes, right.

...

Also, I also read somewhere that MSVC had not yet implemented variadic templates:

Variadic templates are implemented in the updated MSVC 11 toolchain ("Microsoft Visual C++ Compiler Nov 2012 CTP"). http://www.microsoft.com/en-us/download/details.aspx?id=35515

Larry Evans

13 Jun 13 Jun

12:27 p.m.

On 06/12/13 04:54, Igor R wrote:

...

...
I probably jumped to the conclusion that "template argument number" meant "number of template instantiations". Sorry for that mistake. I guess "template argument number" means the maximum arity of any template. IOW, if "template argument number" was 5, then any template taking over 5 arguments would cause the MSVC10/11 compiler to diagnose an error. IOW:

tuple

would be OK, but:

tuple

would cause MSV10/11 to issue a compiler error. Is that right?

Yes, right.

...
Also, I also read somewhere that MSVC had not yet implemented variadic templates:

Variadic templates are implemented in the updated MSVC 11 toolchain ("Microsoft Visual C++ Compiler Nov 2012 CTP"). http://www.microsoft.com/en-us/download/details.aspx?id=35515

Thanks for the link. Now with variadic templates, can a tuple with > 64 elements be created? IOW, would the following: template struct tuple; tuple*pt=0; compile? FWIW, I used Christopher's variadic fusion library to compile a 300 element tuple. It took a while, but it compiled. Also, timing results showing the compile time vs tuple size with gcc4.8 are: TUPLE_SIZE= 10 1.97 0.19 2.14 TUPLE_SIZE= 50 2.45 0.16 2.59 TUPLE_SIZE= 100 3.49 0.32 3.90 TUPLE_SIZE= 200 7.90 0.57 8.85 TUPLE_SIZE= 300 15.93 0.96 17.09 The timing results were produce on ubuntu with: TIME.cmd=time --format '%U %S %e' I think this gives: user-time system-time elapsed-time It also appears that when TUPLE_SIZE >=100, compile time goes up linearly. The code compiled was: typedef slim::vector < int_value< 0 > , int_value< 1 > , int_value< 2 > , int_value< 3 > . . .

...

tuple_t; int run(tuple_t const&a_tuple) { int sum=0; sum+=slim::at_c< 0 >(a_tuple).value; sum+=slim::at_c< 1 >(a_tuple).value; sum+=slim::at_c< 2 >(a_tuple).value; sum+=slim::at_c< 3 >(a_tuple).value; . . . sum+=slim::at_c< 49 >(a_tuple).value; return sum; } and was generated by the attached python file with TUPLE_SIZE as the argument. -regards, Larry

Slava

14 Jun 14 Jun

9:39 a.m.

On Thu, 13 Jun 2013 14:27:30 +0200, Larry Evans wrote:

...

It also appears that when TUPLE_SIZE >=100, compile time goes up linearly.

Take a look at memory usage either, as soon as it starts swapping, it's far from linear. Your peers will curse you for this code. CDT indexer (eclipse) takes a lot time and memory on such constructs. The size of the produced binary (due to debug info for all these small inlined internal helper functions) must not be forgotten. It's hard to justify half-Gb executable to the colleagues. -- Slava

Larry Evans

2:08 p.m.

On 06/14/13 04:39, Slava wrote:

...

On Thu, 13 Jun 2013 14:27:30 +0200, Larry Evans wrote:

...
It also appears that when TUPLE_SIZE >=100, compile time goes up linearly.

I think that linear remark was wrong. It goes up by a factor of about 2 between 100 and 200 and then by another factor of 2 between 200 and 300. That's exponential IIRC :( Of course I could be misunderstanding again. It happens :(

...

Take a look at memory usage either, as soon as it starts swapping, it's far from linear. Your peers will curse you for this code. CDT indexer (eclipse) takes a lot time and memory on such constructs. The size of the produced binary (due to debug info for all these small inlined internal helper functions) must not be forgotten. It's hard to justify half-Gb executable to the colleagues.

-- Slava

Ah! Thanks for the tip. I had no idea that was the case. Would spirit's fusion do any better? Maybe a better question is would there be any reason for such large tuples. I asked Rodrigo that question in another post; however, I've not heard back from him. --Larry

Larry Evans

6:38 p.m.

On 06/14/13 09:08, Larry Evans wrote:

...

On 06/14/13 04:39, Slava wrote:

...
On Thu, 13 Jun 2013 14:27:30 +0200, Larry Evans wrote:

...
It also appears that when TUPLE_SIZE >=100, compile time goes up linearly.

I think that linear remark was wrong. It goes up by a factor of about 2 between 100 and 200 and then by another factor of 2 between 200 and 300. That's exponential IIRC :( Of course I could be misunderstanding again. It happens :(

...
Take a look at memory usage either, as soon as it starts swapping, it's far from linear. Your peers will curse you for this code. CDT indexer

Yes. Looking at the performance monitor as it was compiling TUPLE_SiZE=500 showed almost all the memory was used and, IIRC, about 40% of swap.

...

...
(eclipse) takes a lot time and memory on such constructs. The size of the produced binary (due to debug info for all these small inlined internal helper functions) must not be forgotten. It's hard to justify half-Gb executable to the colleagues.

-- Slava [snip]

Using the slim::accumulate template instead of using python to generate the summation code doesn't make much difference: tuple_size compile_time 10 2.01 50 3.37 100 6.26 200 18.51 300 43.79 400 109.28 500 195.42 Attached is the driver. I did try to eyeball a fit with qtiplot which showed: compile_time = 12*2^(max(0,tuple_size-100)/100) looked quite similar at tuple_size>=100. --Larry

Rodrigo Madera

15 Sep 15 Sep

2:39 a.m.

Sorry for not writing back at this thread. It's been very informative. So thanks! Larry, I had this issue for a hypothetical matrix computing problem, but nothing much practical. I'm sticking to smaller vectors now. Best regards, Rodrigo On Fri, Jun 14, 2013 at 3:38 PM, Larry Evans wrote:

...

On 06/14/13 09:08, Larry Evans wrote:

...
On 06/14/13 04:39, Slava wrote:

...
On Thu, 13 Jun 2013 14:27:30 +0200, Larry Evans wrote:

...
It also appears that when TUPLE_SIZE >=100, compile time goes up linearly.

I think that linear remark was wrong. It goes up by a factor of about 2 between 100 and 200 and then by another factor of 2 between 200 and 300. That's exponential IIRC :( Of course I could be misunderstanding again. It happens :(

...
Take a look at memory usage either, as soon as it starts swapping, it's far from linear. Your peers will curse you for this code. CDT indexer

Yes. Looking at the performance monitor as it was compiling TUPLE_SiZE=500 showed almost all the memory was used and, IIRC, about 40% of swap.

...
...
(eclipse) takes a lot time and memory on such constructs. The size of the produced binary (due to debug info for all these small inlined internal helper functions) must not be forgotten. It's hard to justify half-Gb executable to the colleagues.

-- Slava [snip]

Using the slim::accumulate template instead of using python to generate the summation code doesn't make much difference:

tuple_size compile_time 10 2.01 50 3.37 100 6.26 200 18.51 300 43.79 400 109.28 500 195.42

Attached is the driver.

I did try to eyeball a fit with qtiplot which showed:

compile_time = 12*2^(max(0,tuple_size-100)/100)

looked quite similar at tuple_size>=100.

--Larry

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Larry Evans

12 Jun 12 Jun

10:56 a.m.

On 06/10/13 13:40, Rodrigo Madera wrote:

...

Hello,

What is the maximum Boost Fusion tuple param count?

I was thinking of using it for about 200-300 elements.

Hi Rodrigo, 200-300 elements seem like an awful lot! I'm curious about what application would require such a large tuple. Could you tell us for what you were planning to use the large tuple? TIA. -regards, Larry

4090

Age (days ago)

4187

Last active (days ago)

List overview

Download

11 comments

4 participants

participants (4)

Igor R
Larry Evans
Rodrigo Madera
Slava