On 19 December 2013 13:26, Andrey Semashev wrote:

When you pull the superproject, it only checks out the submodules at the specific revision that, I believe, is updated manually now by someone with the write access to the superproject.

I have a script which updates it, but I'm running it manually as it's not quite ready to be a cron job. If it's not too much for the server it should be updating automatically soon.

On 19 December 2013 14:11, Sergey Cheban wrote:

On 19.12.2013 17:32, Daniel James wrote:

...

...

When you pull the superproject, it only checks out the submodules at the specific revision that, I believe, is updated manually now by someone with the write access to the superproject.

I have a script which updates it, but I'm running it manually as it's not quite ready to be a cron job. If it's not too much for the server it should be updating automatically soon.

May be, it is better to use github's service hooks instead of cron to initiate the update?

https://help.github.com/articles/post-receive-hooks

Maybe, but it would introduce concurrency issues (if two modules are pushed to in quick succession). If I get it working in a cron job, I can think about hooks later. FWIW I am using a hook to update the website.

On 20 December 2013 00:39, Cox, Michael wrote:

On Thu, Dec 19, 2013 at 7:25 AM, Daniel James wrote:

...

Maybe, but it would introduce concurrency issues (if two modules are pushed to in quick succession). If I get it working in a cron job, I can think about hooks later. FWIW I am using a hook to update the website.

Once you have more than one git repository (superproject and submodules), by definition, you *will* have concurrency issues no matter how you do it. It's just the size of the window of vulnerability you can try to minimize.

Guess why I'm cautious about running the script automatically. Anyway, I can linearise updates to submodules using the github events api. Which means I only have to deal with concurrent updates of the superproject which is an easier problem. What's tricky is dealing with concurrent changes to local state. There are obviously strategies to deal with that but if I can guarantee that there's a reasonable gap between each run of the script life is easier. It also means I don't have to maintain over 100 hooks.

...

BTW I have every library set explicitly to either "develop" or "master".

IMO for libraries that you're not working on the best thing is to use whatever the superproject uses and not bother with branches. We might set the submodules to a different commit for some reason, so you could end up developing against the wrong version of a module.

OK, in other words basically ignore the issue? The danger of this is breakages - for example Boost.Build has completely stopped working for me because the super-project was updated but the subproject was not. A manual update of the boost.build subproject fixed things, but I suspect not everyone would realize that was what was needed. Whatever I suspect folks (well this one at least!) will need some very clear guidance on this, Thanks, John.

Peter A. Bigot wrote:

git pull git submodule update

That's what I do, too. This puts everything (back) into a detached head, so if I want to do work on, for example, smart_ptr, I have to then manually switch it to "devel" again: cd libs/smart_ptr git checkout devel but it's not that much of a bother, and the whole procedure can in principle be automated with a script.

On 12/19/2013 09:05 AM, Edward Diener wrote:

On 12/19/2013 8:00 AM, John Maddock wrote:

...

OK.... so I do a

Git pull

at the top level (note: using TortoiseGit), and I see lots of messages in the log indicating which libraries have updates, and judging by the time taken they seem to be downloaded (?), but I don't actually see any changes in the libraries themselves.

What are reasons for ever 'pulling' the superproject ?

Without trying to find what it's intended to mean for Boost, in normal practice this means: Update to identify the versions of each submodule that are confirmed to interoperate correctly at the head of superproject branch that's being pulled. In other words, the latest successful results of integration testing. "git submodule update" is subsequently required to actually change the contents of the submodule working spaces to contain those versions. It's probably a separate step because that makes it easier to deal with conflicts in specific submodules that were locally modified. Peter

...

"git submodule update" is subsequently required to actually change the contents of the submodule working spaces to contain those versions. It's probably a separate step because that makes it easier to deal with conflicts in specific submodules that were locally modified.

I assume "git submodule update" is done at the superproject level. But as other have said, this puts everything back into detached state so that one needs to checkout whatever branches one was working on before again. Ugh !

Ugh indeed. What happens to uncommitted changes in that case? John.

On Thu, 19 Dec 2013 08:42:54 -0800, Edward Diener wrote:

On 12/19/2013 10:27 AM, Peter A. Bigot wrote:

...

On 12/19/2013 09:05 AM, Edward Diener wrote:

...

On 12/19/2013 8:00 AM, John Maddock wrote:

...

OK.... so I do a

Git pull

at the top level (note: using TortoiseGit), and I see lots of messages in the log indicating which libraries have updates, and judging by the time taken they seem to be downloaded (?), but I don't actually see any changes in the libraries themselves.

What are reasons for ever 'pulling' the superproject ?

Without trying to find what it's intended to mean for Boost, in normal practice this means: Update to identify the versions of each submodule that are confirmed to interoperate correctly at the head of superproject branch that's being pulled.

In other words, the latest successful results of integration testing.

"git submodule update" is subsequently required to actually change the contents of the submodule working spaces to contain those versions. It's probably a separate step because that makes it easier to deal with conflicts in specific submodules that were locally modified.

I assume "git submodule update" is done at the superproject level. But as other have said, this puts everything back into detached state so that one needs to checkout whatever branches one was working on before again. Ugh !

I don't know if this might help, but this was mentioned before on this list: http://www.vogella.com/articles/Git/article.html#submodules "41.4. Tracking branches with submodules Since tracking of branches is a very common requirement, Git added the option to track a certain branch in its 1.8.2 release. ..." As opposed to "41.3. Tracking commits".

On 12/19/2013 05:42 PM, Edward Diener wrote:

On 12/19/2013 10:27 AM, Peter A. Bigot wrote:

...

On 12/19/2013 09:05 AM, Edward Diener wrote:

...

On 12/19/2013 8:00 AM, John Maddock wrote:

...

OK.... so I do a

Git pull

at the top level (note: using TortoiseGit), and I see lots of messages in the log indicating which libraries have updates, and judging by the time taken they seem to be downloaded (?), but I don't actually see any changes in the libraries themselves.

What are reasons for ever 'pulling' the superproject ?

Without trying to find what it's intended to mean for Boost, in normal practice this means: Update to identify the versions of each submodule that are confirmed to interoperate correctly at the head of superproject branch that's being pulled.

In other words, the latest successful results of integration testing.

"git submodule update" is subsequently required to actually change the contents of the submodule working spaces to contain those versions. It's probably a separate step because that makes it easier to deal with conflicts in specific submodules that were locally modified.

I assume "git submodule update" is done at the superproject level. But as other have said, this puts everything back into detached state so that one needs to checkout whatever branches one was working on before again. Ugh !

This can be automated in a number of ways, including use of git built in features such as: $git submodule update --rebase --rebase This option is only valid for the update command. Rebase the current branch onto the commit recorded in the superproject. If this option is given, the submodule’s HEAD will not be detached. If a merge failure prevents this process, you will have to resolve these failures with git-rebase(1). If the key submodule.$name.update is set to rebase, this option is implicit. -- Bjørn

On Dec 19, 2013 3:01 PM, "Bjørn Roald" wrote:

On 12/19/2013 02:00 PM, John Maddock wrote:

...

OK.... so I do a

Git pull

at the top level (note: using TortoiseGit), and I see lots of messages in the log indicating which libraries have updates, and judging by the time taken they seem to be downloaded (?), but I don't actually see any changes in the libraries themselves.

BTW I have every library set explicitly to either "develop" or "master".

If I do a "git pull" on an individual library then the changes do come through OK.

So what's the right way to do this? I assume it's not a "submodule update" as that seems to leave everything in a detached head state which I take it is a bad idea?

I'm also seeing a lot of negative comments about Git submodules on the web, for example:

http://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt... ,

...

seems like there's a lot of potential to shoot yourself in the foot/head/chest here?

I don't think it is hard to agree that git submodule it not a perfect tool, but which available alternative is perfect?

I have the feeling it is because it is not as simple of a problem as it seems at first and it has taken a long time for the tools to mature to current level. Hopefully there are still potential for improvements.

I have seen naive attempts to do what submodule attempts to do, "the right way", that has failed completely in improving the overall situation. Champions of such attempts often get too much invested interest in their solution to admit its failure. If I should suggest something it would be to use submodules but provide firm guidelines and/or scripts help the developers with simple ways to do it right and avoid the pitfalls.

No amount of scripting is going yo help those of us not wanting to use the git cli. And this is something I mentioned early on. If something past basic git functionality is needed as a regular course of use the move to git will fail. I saw an article some time ago about a git feature that should be used instead of sub modules.. but I can't find it right now. Perhaps someone can find that alternative arrangement? Rene.

On 12/19/2013 11:14 PM, Rene Rivera wrote:

No amount of scripting is going yo help those of us not wanting to use the git cli. And this is something I mentioned early on. If something past basic git functionality is needed as a regular course of use the move to git will fail.

You have point, however I am not sure the GUI you choose know nothing of how to help you use submodule and avoid its pitfalls, In any case, since only a few will have to do commits in the super project that changes submodule commits, and that may even be more or less completely automated, I am not sure these advertised problems with submodules will actually manifest themselves in Boost.

I saw an article some time ago about a git feature that should be used instead of sub modules.. but I can't find it right now. Perhaps someone can find that alternative arrangement?

There was some talk on the list on git subtree which are now standard command in git, but I doubt it is a good alternative for Boost. -- Bjørn

On 20 December 2013 03:15, Rene Rivera wrote:

In the git.py module I wrote for testing I deal with submodules for fetching mostly correctly.. But even it doesn't work for fetching Boost as the content itself confuses it into throwing exceptions. So now I'm faced with writing a wrapper around the git cli for testing.. And feeling like I don't have any idea what the proper way to fetch the correct stuff from github is. But I'll post about that in another thread.

Wrapping the cli is probably the best strategy, other implementations never quite seem to match it. FWIW here's the script I quickly wrote to mirror the repos for the documentation build: https://github.com/danieljames/boost-build-docs/blob/master/git-mirror.py

On 12/20/2013 6:15 AM, Klaim - Joël Lamotte wrote:

[sidenote about guis] I've used TortoiseGit for some time but I don't like the fact that they keep trying to emulate TortoiseSVN. TortoiseHg does it (for mercurial) in a better way by just keeping the basic principles of TortoiseSVN but not keeping the SVN-induced interface of TortoiseSVN. It means that the best way to work with DSVC is often to have a separate interface to see the overall tree of commits and other repository informations, so they provide a "workbench".

These days I only use SourceTree for both git and hg. It's cross platform, it's almost as useful as the TortoiseHg workbench and it manage as many repositories (and subrepos) as you want in only one interface.

According to what I see it does not run on Linux. But it looks like a pretty nice Windows alternative.

Other pros: it allowed me to understand git flow better, it provides pre-configured terminal in the context of a repository so that you can do recipes the hard way if you want, and it can (if you chose to do it this way) use an embedded hg/git version instead of the one installed on the OS. Cons: it's often slow to respond, but last versions have been improved a lot on this side. I also have some issues with some details of the ergonomy but from exchanges with the devs, I apparently have an alien way of using some of the features.

Basically, if TortoiseGit don't feels right and you don't want to use command line all the time, I suggest trying SourceTree. I have the boost repo in SourceTree right now, but didn't tried forking etc. yet, so I have no idea if it will manage the load manipulating 150+ submodules.

I'm studying/trying other tools too for use in cross-plateform contexts, so if you have other suggestiions of tools, I'm interested.

There is also SmartGit ( http://www.syntevo.com/smartgithg ), which is free for non-commercial use. I think TortoiseGit works pretty well myself but it is a bit under-documented, with many GUI screens not providing an explanation of all the options. Also their online Google group forum is not nearly as active as the TortoiseSVN one.

[/sidenote]

On Thu, Dec 19, 2013 at 5:14 PM, Rene Rivera wrote:

No amount of scripting is going yo help those of us not wanting to use the git cli. And this is something I mentioned early on. If something past basic git functionality is needed as a regular course of use the move to git will fail.

TortoiseGit seems to work well in a submodule environment, although I've only used the "git submodule ..." equivalents a few times. I don't have any experience with the other Git GUI's, so can't speak for them. --Beman

On 12/22/2013 11:08 AM, Paul A. Bristow wrote:

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Beman Dawes Sent: Saturday, December 21, 2013 2:48 AM To: Boost Developers List Subject: Re: [boost] [git] How the blazes are you supposed to update all the submodules?

On Thu, Dec 19, 2013 at 5:14 PM, Rene Rivera wrote:

...

No amount of scripting is going yo help those of us not wanting to use the git cli. And this is something I mentioned early on. If something past basic git functionality is needed as a regular course of use the move to git will fail.

TortoiseGit seems to work well in a submodule environment, although I've only used the "git submodule ..." equivalents a few times. I don't have any experience with the other Git GUI's, so can't speak for them.

As someone who has been hiding his head under a blanket up till now (half hoping it would all go away ;-)

I too very, very much trust that it will be possible to do all the routine things using a GUI, and that it will be fully documented using at least one GUI.

Tortoise GIT has worked for me locally, so I'd favour using that for a start, but guidance from those with more experience will of course continue be helpful.

A number of the Tortoise Git dialogs do not give an explanation in the help for some of the GUI options. This is sometimes supplemented in the help by suggesting that one look at the equivalent git command which the GUI dialog implements, but this is often a PITA if it is difficult to map the GUI option to a particular git command line option. TGIT is very decent and has been getting better with each release but it is not as polished as TSVN. One can always of course drop down to the git command line level. As I get better at git, from reading books and documents, this becomes easier.