[multiindex] Ordered index corruption
Hi All, I have a problem with a multiindex ordered_non_unique index becoming corrupted :( I am working with a multiindex that stores class objects of type Individual: class Individual { double I_; double id_; public: double getId() const; double getI() const; void setId(const std::string id); void setI(const double I); } boost::multi_index_container< Individual, indexed_by< random_access< tag<bySeq> >, hashed_unique< tag<byId>, const_mem_fun<Individual,string,&Individual::getId> >, ordered_non_unique<tag<byI>, const_mem_fun<Individual,double,&Individual::getI> > > > I have an algorithm that works on these data, using Individual::I to calculate a summary value across the whole dataset. An Individual i is then chosen at random, and i.I updated using the multiindex::modify method with a functor: struct modifyI { modifyI(double newI) : newI_(newI) {} void operator()(Individual& i) { i.setI(newI_); } private: double newI_; }; This calculation is then repeated many times. The actual calculation is done in parallel using MPI (Boost::MPI on top of OpenMPI), with the final reduction being broadcast to all processes. This enables each processor to make the same change to its copy of the multiindex. In *theory* this should mean that each process has a complete replicate of the multiindex. However, I am finding that the ordered_non_unique index is becoming corrupted in that the ordering (strict less than on Individual::I) gets messed up for a few (c. 2 or 3) of the elements. Essentially, it appears that small blocks of the index get swapped round. This is not occurring simultaneously on all processes, which is odd because the same multiindex::modify method is called for each process with the same arguments. Can anyone suggest a way I can move forward on trying to debug this? Thanks, Chris -- Dr Chris Jewell Department of Statistics University of Warwick Coventry CV4 7AL UK Tel: +44 (0)24 7615 0778
Chris Jewell <chris.jewell <at> warwick.ac.uk> writes:
Hi All,
Hello Chris,
I have a problem with a multiindex ordered_non_unique index becoming corrupted :(
I am working with a multiindex that stores class objects of type Individual:
class Individual { double I_; double id_; public: double getId() const;
I understand I_ is a std:string rather than a double, right?
double getI() const; void setId(const std::string id); void setI(const double I); }
[...]
I have an algorithm that works on these data, using Individual::I to calculate a summary value across the whole dataset. An Individual i is then chosen at random, and i.I updated using the multiindex::modify method with a functor:
struct modifyI { modifyI(double newI) : newI_(newI) {} void operator()(Individual& i) { i.setI(newI_); } private: double newI_; };
This calculation is then repeated many times. The actual calculation is done in parallel using MPI (Boost::MPI on top of OpenMPI), with the final reduction being broadcast to all processes. This enables each processor to make the same change to its copy of the multiindex. In *theory* this should mean that each process has a complete replicate of the multiindex.
Everything about the scenario you describe looks perfectly normal except this. Are you sure no container instance is accessed simultaneously from two different threads? You can do the following sanity check: wrap each access to a container with a mutex guard and see whether the problem persists. If not, your assumption above is incorrect. If the sanity check unveils nothing I'll ask you for more info on your program. Also, you can turn the so-called invariant-checking mode on: http://www.boost.org/libs/multi_index/doc/tutorial/debug.html#invariant_chec... This will hopefully locate the exact point where corruption occurs, which might help you find out what's going wrong. Best, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Hi Joaquin,
I have a problem with a multiindex ordered_non_unique index becoming corrupted :(
I am working with a multiindex that stores class objects of type Individual:
class Individual { double I_; double id_; public: double getId() const;
I understand I_ is a std:string rather than a double, right?
Actually, id_ is a std::string. I_ is a double.
double getI() const; void setId(const std::string id); void setI(const double I); }
[...]
I have an algorithm that works on these data, using Individual::I to calculate a summary value across the whole dataset. An Individual i is then chosen at random, and i.I updated using the multiindex::modify method with a functor:
struct modifyI { modifyI(double newI) : newI_(newI) {} void operator()(Individual& i) { i.setI(newI_); } private: double newI_; };
This calculation is then repeated many times. The actual calculation is done in parallel using MPI (Boost::MPI on top of OpenMPI), with the final reduction being broadcast to all processes. This enables each processor to make the same change to its copy of the multiindex. In *theory* this should mean that each process has a complete replicate of the multiindex.
Everything about the scenario you describe looks perfectly normal except this. Are you sure no container instance is accessed simultaneously from two different threads?
You can do the following sanity check: wrap each access to a container with a mutex guard and see whether the problem persists. If not, your assumption above is incorrect. If the sanity check unveils nothing I'll ask you for more info on your program.
Yes, I'm absolutely sure that I'm not getting a thread problem.
Also, you can turn the so-called invariant-checking mode on:
http://www.boost.org/libs/multi_index/doc/tutorial/debug.html#invariant_chec...
This will hopefully locate the exact point where corruption occurs, which might help you find out what's going wrong.
I realised that I had a misbehaving function that was writing garbage to the multiindex, and appeared to be causing the corruption. I've yet to sort this out (for now, I've abandoned the function) because I've been busy doing other things, but I'll return to it soon and post some contextual info. Sorry for wasting time! I hope it's interesting that this phenomenon is occurring due to garbage being written...? Thanks for your help! Chris
participants (5)
-
Chris Jewell -
Joaquin M Lopez Munoz