asio's io_service uses too many file descriptors

newer
[program_options] Short program...

pete＠mu.org

15 Aug 2007 15 Aug '07

5:28 a.m.

I posted this under a less suitable subject line a week ago, but I got no response. I hope the above live will trigger some help. I 've been doinking around with asio for a few weeks, and I've run into an issue I can't seem to wrap my head around. I've been playing with some code that originally was meant to pound a server with a large amount of packets for benchmarking purposes. It uses threads to launch hundreds of threads per second. When I replaced the old socket code with asio, and used a single io_service with run being called within each thread, I run out of file descriptors very quickly, and if I use a single io_service within each thread it's the same thing but the performance seems worse. Needless to say, this seems like a real limitation and I'm hoping I'm just being limited by my own misunderstanding, because I'm really enjoying the portability of asio. The docs that are there seem great, but there seems to be a lot more the library can do that I am missing out on, and I think the docs seem like they are not mentioning a lot of what asio an do. Can someone point me in the right direction for launching 1-to-N threads with a single send_to and a single receive_from within each thread that doesn't take the hit I'm seeing? It seems to me, that it's obvious that the io-service design is the difference, because I don't get the issue at all when I use plain old sockets, and the OS thread limit is then the only limitation. Any help would be greatly appreciated. FYI, I've done most of my experimenting on Linux and OSX. I can send some code samples if needed, but I think the problem should be apparent if anyone else has tried to use asio with large amounts of threads. Thanks, Pete

Show replies by date

Manuel Jung

15 Aug 15 Aug

7:52 a.m.

Hi, I'm not sure if i understood exactly what are you doing, but take a look at this: You don't need many io_service's. You just need one which runs a new thread, if a new connection to the server is made. So the io_service object is sitting around all time waiting (because you have called async_accept)- if a client connects, you can start a thread for this client and use sync network operations or you can process a bit of data and then go into another async operation, which calls the next function, if the data has been received or send. One or the other way you just need one io_service object, because shortly after a connection has been accepted, the connection is processed in another thread and the io_service object is runned again. As far as i can tell it just makes sense to run different io_service objects, if you have a multicore system and want to use all cores to accept connections. This was discussed some time ago on the mailing list and i think they wanted to take a new example into asio to show, how its done. Some exapmles for a server like i have described it are included in asio. e.g. the http server example. Did this answer you question? Kind Regards Manuel Jung pete@mu.org wrote:

...

I posted this under a less suitable subject line a week ago, but I got no response. I hope the above live will trigger some help.

I 've been doinking around with asio for a few weeks, and I've run into an issue I can't seem to wrap my head around.

I've been playing with some code that originally was meant to pound a server with a large amount of packets for benchmarking purposes. It uses threads to launch hundreds of threads per second.

When I replaced the old socket code with asio, and used a single io_service with run being called within each thread, I run out of file descriptors very quickly, and if I use a single io_service within each thread it's the same thing but the performance seems worse. Needless to say, this seems like a real limitation and I'm hoping I'm just being limited by my own misunderstanding, because I'm really enjoying the portability of asio.

The docs that are there seem great, but there seems to be a lot more the library can do that I am missing out on, and I think the docs seem like they are not mentioning a lot of what asio an do.

Can someone point me in the right direction for launching 1-to-N threads with a single send_to and a single receive_from within each thread that doesn't take the hit I'm seeing?

It seems to me, that it's obvious that the io-service design is the difference, because I don't get the issue at all when I use plain old sockets, and the OS thread limit is then the only limitation.

Any help would be greatly appreciated. FYI, I've done most of my experimenting on Linux and OSX.

I can send some code samples if needed, but I think the problem should be apparent if anyone else has tried to use asio with large amounts of threads.

Thanks,

Pete

pete＠mu.org

3:55 p.m.

On Wed, Aug 15, 2007 at 09:52:38AM +0200, Manuel Jung wrote:

...

As far as i can tell it just makes sense to run different io_service objects, if you have a multicore system and want to use all cores to accept connections. This was discussed some time ago on the mailing list and i think they wanted to take a new example into asio to show, how its done.

Thanks a lot for the reply. However, I think I'm probably miscommunicating the issue. The docs are pretty specific that you can either have a global io_service and then have each thread call run from within each on, or you can have a an io_service for each thread. I have tried both. Both have the same issue. If I use standard low level sockets, my app runs very fast and concurrency pays off, but with asio it's a dog that runs our of resources. My test app is very simple it just tries to do as many hits on a server within each thread with a single packet as possible and then reads back the single reply. However, it doesn't matter, because I'm using the io service, it limits me from making new sockets because it's running out of file descriptors. The connections are never made because it can't even create a socket. On my standard 32bit linux system, it can't do more than 25 or 30 threads before asio honks. All I want to do is launch hundreds of threads to make these connections and asio doesn't seem to permit that sort of abuse. As this type of thing might not be done normally, I can see why no one else might of seen it, but certainly if a standard posix implementation can do it, then it would seem asio should allow for that too. ie., isn't there a way to just allow for me to use a bunch of send_to's and recv_from's without the resouces running out so fast? Does that make sense? Thanks, Pete

Darren Garvey

5:09 p.m.

Hey Pete, On 15/08/07, pete@mu.org wrote:

...

The docs are pretty specific that you can either have a global io_service and then have each thread call run from within each on, or you can have a an io_service for each thread.

IIUC, you can mix and match too (see the HTTP3 example, which uses an io_service pool). In your case, having a pool of io_services, each being run in a pool of threads might help you. I have tried both. Both have the same issue. If I use standard low level

...

sockets, my app runs very fast and concurrency pays off, but with asio it's a dog that runs our of resources.

Are you using Linux 2.4? According to the docs, you are limited to FD_SETSIZE number of sockets. Can you check what that is defined as on your system? You shouldn't run into this problem a newer Linux kernel, as that uses epoll, which apparently doesn't have the same limitations. <snip>

...

All I want to do is launch hundreds of threads to make these connections

...

and asio doesn't seem to permit that sort of abuse.

Asio should allow that sort of abuse. Can you provide more details about your system? As an aside, if you fork() before running your 'main loop', you can get around the FD_SETSIZE limit (which is a per-process one). Regards, Darren

pete＠mu.org

5:50 p.m.

Hey Darren, On Wed, Aug 15, 2007 at 06:09:43PM +0100, Darren Garvey wrote:

...

On 15/08/07, pete@mu.org wrote: IIUC, you can mix and match too (see the HTTP3 example, which uses an io_service pool). In your case, having a pool of io_services, each being run in a pool of threads might help you.

I used some of that for my original model when moving to asio from native sockets. However, I think I just found my problem. It appears I'm an idiot! Doh! I was having problems when porting it originally, so I started using log4cxx from the apache folks for doing some thread safe logging to debug the issue. I think it's the log4cxx that is actually causing the problem now. It's using up my FD's it appears, as I don't have any issues when I remove the logging. The exceptions I've been looking for are all from boost::asio and not from log4cxx, so I think I was blinded by that. When I removed all the threaded logging, the problem has vanished---I think. Sorry for being a weenie. Thanks for your response! It helped me work out the issue by trying to answer your questions. Have a good one, Pete

...

I have tried both. Both have the same issue. If I use standard low level

...
sockets, my app runs very fast and concurrency pays off, but with asio it's a dog that runs our of resources.

Are you using Linux 2.4? According to the docs, you are limited to FD_SETSIZE number of sockets. Can you check what that is defined as on your system? You shouldn't run into this problem a newer Linux kernel, as that uses epoll, which apparently doesn't have the same limitations.

<snip>

...
All I want to do is launch hundreds of threads to make these connections

...
and asio doesn't seem to permit that sort of abuse.

Asio should allow that sort of abuse. Can you provide more details about your system?

As an aside, if you fork() before running your 'main loop', you can get around the FD_SETSIZE limit (which is a per-process one).

Regards, Darren

...

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

6316

Age (days ago)

6316

Last active (days ago)

List overview

Download

4 comments

3 participants

participants (3)

Darren Garvey
Manuel Jung
pete＠mu.org