[BGL] visualizing large graphs
Hi, although this might seem a bit off-topic, I think it is an important topic and might be very well interesting for some of us. The boost graph library provides methods to export graphs e.g. for the use with graphviz. However, this is a rather "one-way" solution. Besides that, it is difficult to handle large graphs with graphviz. Therefore, I was wondering if there have been some recent developments here? I would especially be interested in a solution that allows the direct visualisation of a graph (even previously serialized), possibly integrating ways to filter out some of the data in order to be able to visualize the graph in some way. It is clear to me that visualizing arbitrarily large graphs will be limited by physical limitations mostly (RAM etc.). But I would find it very convenient to at least load the graph, set some filters, so that I don't need to do this seperatly via a C++-program. Actually, I am writing my graph to a format that can be visualized by Cytoscape. Unfortunately, this suffers from large limitations on the size of the graph (number of edges and nodes). I am looking forward to hear your ideas about this and possible suggestions about libraries or programs that might achieve this goal. Thank you. Best regards, Cedric
Hi Cedric, I have been trying to use a program called Tulip to visualise my graphs, it claims to work up to millions of nodes, but as yet I still get crashes for relatively small graphs. My main problem has been that all my machines with enough RAM are remote and the x-connection is too slow. I have put some more RAM in my desktop but now the graphics card seems to struggle. Waiting for a new graphics card ... Tulip does support batch mode, so I could run on a big memory machine and output hardcopy, but I haven't had a chance to really learn it interactively yet. I'll let you know if I have any success. How big a graph can you look at with cytoscape? I haven't tried that. Best, Adam. -- Dr Adam Spargo High Performance Assembly Group email: aws@sanger.ac.uk Wellcome Trust Sanger Institute Tel: +44 (0)1223 834244 x7728 Hinxton, Cambridge CB10 1SA Fax: +44 (0)1223 494919 On Wed, 29 Sep 2010, Cedric Laczny wrote:
Hi,
although this might seem a bit off-topic, I think it is an important topic and might be very well interesting for some of us. The boost graph library provides methods to export graphs e.g. for the use with graphviz. However, this is a rather "one-way" solution. Besides that, it is difficult to handle large graphs with graphviz. Therefore, I was wondering if there have been some recent developments here? I would especially be interested in a solution that allows the direct visualisation of a graph (even previously serialized), possibly integrating ways to filter out some of the data in order to be able to visualize the graph in some way. It is clear to me that visualizing arbitrarily large graphs will be limited by physical limitations mostly (RAM etc.). But I would find it very convenient to at least load the graph, set some filters, so that I don't need to do this seperatly via a C++-program. Actually, I am writing my graph to a format that can be visualized by Cytoscape. Unfortunately, this suffers from large limitations on the size of the graph (number of edges and nodes). I am looking forward to hear your ideas about this and possible suggestions about libraries or programs that might achieve this goal. Thank you.
Best regards,
Cedric _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
On Wednesday, 29. September 2010 11:36:03 Adam Spargo wrote:
Hi Cedric, I have been trying to use a program called Tulip to visualise my graphs, it claims to work up to millions of nodes, but as yet I still get crashes for relatively small graphs. My main problem has been that all my machines with enough RAM are remote and the x-connection is too slow. I have put some more RAM in my desktop but now the graphics card seems to struggle. Waiting for a new graphics card ... Tulip does support batch mode, so I could run on a big memory machine and output hardcopy, but I haven't had a chance to really learn it interactively yet.
I'll let you know if I have any success.
Thank you, I will give it a try
How big a graph can you look at with cytoscape? I haven't tried that.
If I remember correctly, you can load graphs with > 10k edges (no clue about the number of vertices, sorry) into cytoscape, but when you try to create an actual representation of that data, even with a "decent" 4GB RAM, it crashes. But I think that this is rather related due to the fact that cytoscape (at least my installation) is based on Java and I get an Buffer-overflow. And I could imagine that for the special purposes in your case at the WTSI, you are handling way bigger graphs ;)
Best,
Adam.
Best, Cedric
-- Dr Adam Spargo High Performance Assembly Group email: aws@sanger.ac.uk Wellcome Trust Sanger Institute Tel: +44 (0)1223 834244 x7728 Hinxton, Cambridge CB10 1SA Fax: +44 (0)1223 494919
On Wed, 29 Sep 2010, Cedric Laczny wrote:
Hi,
although this might seem a bit off-topic, I think it is an important topic and might be very well interesting for some of us. The boost graph library provides methods to export graphs e.g. for the use with graphviz. However, this is a rather "one-way" solution. Besides that, it is difficult to handle large graphs with graphviz. Therefore, I was wondering if there have been some recent developments here? I would especially be interested in a solution that allows the direct visualisation of a graph (even previously serialized), possibly integrating ways to filter out some of the data in order to be able to visualize the graph in some way. It is clear to me that visualizing arbitrarily large graphs will be limited by physical limitations mostly (RAM etc.). But I would find it very convenient to at least load the graph, set some filters, so that I don't need to do this seperatly via a C++-program. Actually, I am writing my graph to a format that can be visualized by Cytoscape. Unfortunately, this suffers from large limitations on the size of the graph (number of edges and nodes). I am looking forward to hear your ideas about this and possible suggestions about libraries or programs that might achieve this goal. Thank you.
Best regards,
Cedric _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 09/29/2010 04:36 AM, Adam Spargo wrote:
Hi Cedric, I have been trying to use a program called Tulip to visualise my graphs, it claims to work up to millions of nodes, but as yet I still get crashes for relatively small graphs.
That was my experience with Tulip as well.
My main problem has been that all my machines with enough RAM are remote and the x-connection is too slow. I have put some more RAM in my desktop but now the graphics card seems to struggle. Waiting for a new graphics card ... Tulip does support batch mode, so I could run on a big memory machine and output hardcopy, but I haven't had a chance to really learn it interactively yet.
I've been playing a little with OpenGL and large graphs. Without making a huge amount of code, a simple massless electrostatic model can make cool looking layouts and animations. Having that third dimensions makes the layout problem much much easier. It would be interesting to see what you get if you simply squash the result back down to two. Anyone on this list could probably do something in C++ that gets the job done without crashing faster than it took me to finally give up after trying all those student-project-like viewers written in Java.
I'll let you know if I have any success.
Yes, please keep us posted. - Marsh
Hi, after having had some trouble with the installation, I could finally test Tulip. My graph is fairly small at the moment, with about 10k nodes and 20k edges. I'll have to test if it would work in cytoscape, but I doubt it. In that sense, I really like Tulip because it behaves nicely and I can imagine that it will do this also for larger graphs. Although, I don't know how large they really can get. What kind of disappointed me are the layout algorithms in Tulip. Even though there might be some plugin-layouts which I haven't tested, this is really nicely working in cytoscape. Installing e.g. the HistogramView-plugin failed due to not understandable reasons. In total, I would say that Tulip is not really helpful in my case, sadly. Therefore, I hope that some of you may have other solutions or suggestions to the problem. Best, Cedric On Thursday, 30. September 2010 22:52:39 Marsh Ray wrote:
On 09/29/2010 04:36 AM, Adam Spargo wrote:
Hi Cedric, I have been trying to use a program called Tulip to visualise my graphs, it claims to work up to millions of nodes, but as yet I still get crashes for relatively small graphs.
That was my experience with Tulip as well.
My main problem has been that all my machines with enough RAM are remote and the x-connection is too slow. I have put some more RAM in my desktop but now the graphics card seems to struggle. Waiting for a new graphics card ... Tulip does support batch mode, so I could run on a big memory machine and output hardcopy, but I haven't had a chance to really learn it interactively yet.
I've been playing a little with OpenGL and large graphs. Without making a huge amount of code, a simple massless electrostatic model can make cool looking layouts and animations.
Having that third dimensions makes the layout problem much much easier. It would be interesting to see what you get if you simply squash the result back down to two.
Anyone on this list could probably do something in C++ that gets the job done without crashing faster than it took me to finally give up after trying all those student-project-like viewers written in Java.
I'll let you know if I have any success.
Yes, please keep us posted.
- Marsh _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Hi,
after having had some trouble with the installation, I could finally test Tulip.
Did you make it run on their example files without crashing? If so how much RAM/graphics card are you using? It has been a while since I tried it, so I can't remember exactly what my latest issue is on the desktop, I think an overfull buffer message followed by a crash. Did you make any changes to the source (buffer sizes etc) before the build? Thanks, Adam. -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Hi, On Friday, 1. October 2010 10:47:36 Adam Spargo wrote:
Hi,
after having had some trouble with the installation, I could finally test Tulip.
Did you make it run on their example files without crashing? If so how much RAM/graphics card are you using?
No, I tried it with my own network that I wrote to a file using boost::write_graphviz(). Could you please tell me where to find such example files? I would then test it.
It has been a while since I tried it, so I can't remember exactly what my latest issue is on the desktop, I think an overfull buffer message followed by a crash. Did you make any changes to the source (buffer sizes etc) before the build?
No, I didn't change anything. Compiled it, installed it and that's it. I have 4GB RAM here and 512MB graphics card. However, that is nothing special as it dates already 2 years ago and wasn't "premium" or similar at that time either.
Thanks,
Adam.
Best, Cedric
No, I tried it with my own network that I wrote to a file using boost::write_graphviz(). Could you please tell me where to find such example files? I would then test it.
I just tried the examples in the tutorial from the user manual: tulip.labri.fr/userHandbook/userManual.pdf worked ok until it crashed. -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Hi, some time ago I found this graph visualization program http://gephi.org/ that is not so know but seems quite promising. Hope this helps, Bye, Tarcisio Fedrizzi.
Hi, it dates back some time, but I tested gephi also and I experienced performance issues when working with larger graphs. Please note that the graphs I am talking about had no problems running with boost on an average machine. So they may have been large but not really huge graphs. IIRC, gephi also suffered from having too few buffer and even after adjusting the buffer as suggested on their website, it was not performing very well. It might be that the actual version is improved. Some quick tests now indicated no performance problems on loading the graph, which is a very crucial point IMHO :) Thanks for reminding about that program. Best, Cedric On Friday, 1. October 2010 11:06:23 Tarcisio Fedrizzi wrote:
Hi, some time ago I found this graph visualization program http://gephi.org/ that is not so know but seems quite promising.
Hope this helps, Bye, Tarcisio Fedrizzi. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
some time ago I found this graph visualization program http://gephi.org/ that is not so know but seems quite promising.
Thanks, will try it out. -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Hi,
although this might seem a bit off-topic, I think it is an important topic and might be very well interesting for some of us. The boost graph library provides methods to export graphs e.g. for the use with graphviz. However, this is a rather "one-way" solution. Besides that, it is difficult to handle large graphs with graphviz.
I think thats an interesting problem, but also very difficult, also the topology you'd like to display the graph in adds to it. I've been playing around with the layout algorithms in BGL now for quite a while, and would like to know, if they could be suitable for your problem. But Kamada Kawai Spring or Fruchterman Reingold are probably quite resource hungry for larger graphs. But in the concept, all you need is a position Map of your graph. The question should be, is there yet an algorithm in Boost fitting this. Also in my Mind is the thought, if maybe a kind of Map-Reduce like solution could solve your problem. regards, Jens Weller -- Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail
Cedric Laczny wrote:
Hi,
although this might seem a bit off-topic, I think it is an important topic and might be very well interesting for some of us. The boost graph library provides methods to export graphs e.g. for the use with graphviz. However, this is a rather "one-way" solution. Besides that, it is difficult to handle large graphs with graphviz.
I've been able to view graphs(DAG) with at least 1M nodes using graphviz with SVG output displayed in a Qt QSVGWidget without problem and in under 1 second on windows XP with the /3GB switch and 4GB ram. The majority of time is in Qt's svg widget interpretation. Any memory limitation were due the the lame design of the underlying node structure, which was replicating lot's of redundant and unnecessary data. I'm not sure what you mean by "one-way". The above svg approach was good enough for my use, but it would have been trivial to replace svg with Qt graphics items, allowing the user to click on a node or edge and have access to the underlying graph data, update the layout, refresh the screen in a reasonable amount of time. Jeff
participants (2)
-
Adam Spargo -
Cedric Laczny -
Jeff Flinn -
Jens Weller -
Marsh Ray -
Tarcisio Fedrizzi