[GitHelp] (Mis-)Understanding merge

I was following Modular Boost document "Establishing a merge point after SVN Conversion" (https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint) and everything appeared to go well. I did my 'merge', 'commit', and 'push' specified at the end of the document and things seemed pretty good so I was confident I had established a correct merge point. Knowing that the latest 'develop' had been pushed and tested I determined to update 'master' with the latest 'develop' prior to pushing it. So I did a merge from 'develop' to 'master' and after carefully fixing a few minor conflicts in favor of the 'develop' versions the merge was successful. Yet when I looked at the files involved I noticed that a file which was in 'develop' was not in 'master'. How can this be ? If I merge from 'develop' to 'master' are not any files in 'develop' which are not in 'master' added automatically to 'master' ? If this is not the case it seems that a 'merge' is not doing what I expect it to do. Am I really expected to have to manually add files to 'master' which are in 'develop' and not in 'master' prior to the merge, or I have I missed some 'merge' functionality that would have done this for me automatically ?

On 26 February 2014 06:50, Edward Diener wrote:
I was following Modular Boost document "Establishing a merge point after SVN Conversion" (https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint) and everything appeared to go well. I did my 'merge', 'commit', and 'push' specified at the end of the document and things seemed pretty good so I was confident I had established a correct merge point.
Knowing that the latest 'develop' had been pushed and tested I determined to update 'master' with the latest 'develop' prior to pushing it. So I did a merge from 'develop' to 'master' and after carefully fixing a few minor conflicts in favor of the 'develop' versions the merge was successful.
Yet when I looked at the files involved I noticed that a file which was in 'develop' was not in 'master'. How can this be ? If I merge from 'develop' to 'master' are not any files in 'develop' which are not in 'master' added automatically to 'master' ? If this is not the case it seems that a 'merge' is not doing what I expect it to do. Am I really expected to have to manually add files to 'master' which are in 'develop' and not in 'master' prior to the merge, or I have I missed some 'merge' functionality that would have done this for me automatically ?
Could the file have been previously added and then removed from master? What were the exact commands you ran? What is the file in question? (It should be possible for everyone else to reproduce the exact same situation and examine what's going on, which is easier than debugging prose.)

On 2/26/2014 2:19 AM, Jonathan Wakely wrote:
On 26 February 2014 06:50, Edward Diener wrote:
I was following Modular Boost document "Establishing a merge point after SVN Conversion" (https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint) and everything appeared to go well. I did my 'merge', 'commit', and 'push' specified at the end of the document and things seemed pretty good so I was confident I had established a correct merge point.
Knowing that the latest 'develop' had been pushed and tested I determined to update 'master' with the latest 'develop' prior to pushing it. So I did a merge from 'develop' to 'master' and after carefully fixing a few minor conflicts in favor of the 'develop' versions the merge was successful.
Yet when I looked at the files involved I noticed that a file which was in 'develop' was not in 'master'. How can this be ? If I merge from 'develop' to 'master' are not any files in 'develop' which are not in 'master' added automatically to 'master' ? If this is not the case it seems that a 'merge' is not doing what I expect it to do. Am I really expected to have to manually add files to 'master' which are in 'develop' and not in 'master' prior to the merge, or I have I missed some 'merge' functionality that would have done this for me automatically ?
Could the file have been previously added and then removed from master?
What were the exact commands you ran? What is the file in question? (It should be possible for everyone else to reproduce the exact same situation and examine what's going on, which is easier than debugging prose.)
It is the preprocessor library. I established the merge point and pushed it. Then I tried to merge 'develop' into 'master'. But one of the files in 'develop', variadic_seq_to_seq.hpp, was missing from 'master' after the merge. I am still trying to understand why this is the case since the merge point should have that file.

On 26 February 2014 06:50, Edward Diener
I was following Modular Boost document "Establishing a merge point after SVN Conversion" (https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint) and everything appeared to go well. I did my 'merge', 'commit', and 'push' specified at the end of the document and things seemed pretty good so I was confident I had established a correct merge point.
Knowing that the latest 'develop' had been pushed and tested I determined to update 'master' with the latest 'develop' prior to pushing it. So I did a merge from 'develop' to 'master' and after carefully fixing a few minor conflicts in favor of the 'develop' versions the merge was successful.
Yet when I looked at the files involved I noticed that a file which was in 'develop' was not in 'master'. How can this be ? If I merge from 'develop' to 'master' are not any files in 'develop' which are not in 'master' added automatically to 'master' ? If this is not the case it seems that a 'merge' is not doing what I expect it to do. Am I really expected to have to manually add files to 'master' which are in 'develop' and not in 'master' prior to the merge, or I have I missed some 'merge' functionality that would have done this for me automatically ?
Assuming you're talking about preprocessor there are a lot of
differences between master and your merge base, when you merge git
won't take these into account - it only deals with changes made after
the last merge. This is the tricky thing about transitioning from
subversion to git, they have different models of merging, subversion
is based on tracking individual changesets, git is based on the
history graph. You could try finding the old changesets that haven't
been merged and cherry pick them. In this case, I found the commit by
doing:
> git log origin/develop --
include/boost/preprocessor/seq/variadic_seq_to_seq.hpp
commit b60a47252d1b87d169bc4eac2b19f6c2916e9983
Author: Paul Mensonides

On 2/26/2014 2:28 AM, Daniel James wrote:
On 26 February 2014 06:50, Edward Diener
wrote: I was following Modular Boost document "Establishing a merge point after SVN Conversion" (https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint) and everything appeared to go well. I did my 'merge', 'commit', and 'push' specified at the end of the document and things seemed pretty good so I was confident I had established a correct merge point.
Knowing that the latest 'develop' had been pushed and tested I determined to update 'master' with the latest 'develop' prior to pushing it. So I did a merge from 'develop' to 'master' and after carefully fixing a few minor conflicts in favor of the 'develop' versions the merge was successful.
Yet when I looked at the files involved I noticed that a file which was in 'develop' was not in 'master'. How can this be ? If I merge from 'develop' to 'master' are not any files in 'develop' which are not in 'master' added automatically to 'master' ? If this is not the case it seems that a 'merge' is not doing what I expect it to do. Am I really expected to have to manually add files to 'master' which are in 'develop' and not in 'master' prior to the merge, or I have I missed some 'merge' functionality that would have done this for me automatically ?
Assuming you're talking about preprocessor
Yes, how did you figure that out ?
there are a lot of differences between master and your merge base, when you merge git won't take these into account - it only deals with changes made after the last merge.
I thought Git was always smart enough to find the merge base and then do a three way merge between the merge base, the commit being merged to and the commit being merged from.
This is the tricky thing about transitioning from subversion to git, they have different models of merging, subversion is based on tracking individual changesets, git is based on the history graph. You could try finding the old changesets that haven't been merged and cherry pick them.
This would theoretically mean that I would have to go one by one through every 'develop' change. Ugh ! I thought Git was supposed to save me from having to do all that tedious work.
In this case, I found the commit by doing:
> git log origin/develop -- include/boost/preprocessor/seq/variadic_seq_to_seq.hpp
commit b60a47252d1b87d169bc4eac2b19f6c2916e9983 Author: Paul Mensonides
Date: Wed Sep 26 04:50:27 2012 added VARIADIC_SEQ_TO_SEQ
[SVN r80704]
You are a genius ! That is the one that did not get added when I merged to 'master' from 'develop'. I still do not understand why Git was not smart enough to realize that.
This gave me the change that need to be cherry picked:
> git cherry-pick b60a47252d1b87d169bc4eac2b19f6c2916e9983
But even after doing that there are still a lot of old changes that need to be merged, mainly in the documentation:
> git diff --stat 5e0422ff9732b0ddb2908f381d58d6241b0d8461 (master) doc/headers.html | 32 ++++++++++++++++---------------- doc/headers/facilities.html | 2 +- doc/headers/facilities/overload.html | 4 ++-- doc/headers/tuple.html | 18 +++++++++--------- doc/headers/tuple/eat.html | 2 +- doc/headers/tuple/elem.html | 2 +- doc/headers/tuple/enum.html | 4 ++-- doc/headers/tuple/rem.html | 4 ++-- doc/headers/tuple/reverse.html | 2 +- doc/headers/tuple/size.html | 4 ++-- doc/headers/tuple/to_array.html | 4 ++-- doc/headers/tuple/to_list.html | 2 +- doc/headers/tuple/to_seq.html | 2 +- doc/headers/variadic.html | 14 +++++++------- doc/headers/variadic/elem.html | 4 ++-- doc/headers/variadic/size.html | 4 ++-- doc/headers/variadic/to_array.html | 4 ++-- doc/headers/variadic/to_list.html | 4 ++-- doc/headers/variadic/to_seq.html | 4 ++-- doc/headers/variadic/to_tuple.html | 4 ++-- doc/ref.html | 36 ++++++++++++++++++------------------ doc/ref/overload.html | 4 ++-- doc/ref/tuple_eat.html | 8 ++++---- doc/ref/tuple_elem.html | 6 +++--- doc/ref/tuple_enum.html | 8 ++++---- doc/ref/tuple_rem.html | 8 ++++---- doc/ref/tuple_rem_ctor.html | 8 ++++---- doc/ref/tuple_reverse.html | 8 ++++---- doc/ref/tuple_size.html | 4 ++-- doc/ref/tuple_to_array.html | 8 ++++---- doc/ref/tuple_to_list.html | 8 ++++---- doc/ref/tuple_to_seq.html | 8 ++++---- doc/ref/variadic_elem.html | 4 ++-- doc/ref/variadic_seq_to_seq.html | 2 +- doc/ref/variadic_size.html | 4 ++-- doc/ref/variadic_to_array.html | 4 ++-- doc/ref/variadic_to_list.html | 4 ++-- doc/ref/variadic_to_seq.html | 2 +- doc/ref/variadic_to_tuple.html | 4 ++-- doc/topics/variadic_macros.html | 7 ++++--- include/boost/preprocessor.hpp | 2 +- include/boost/preprocessor/config/config.hpp | 2 +- 42 files changed, 135 insertions(+), 134 deletions(-)
You could try to track down all the appropriate changes, but IMO the best thing to do is to get develop to a release-worthy point and just overwrite master from develop.
I could do that. If I did do I just push 'develop' to 'origin/master' ? I could also, after having done my local merge, just do a 'git diff master develop' and then update master with appropriate changes. I am probably naive but I still do not understand why the 'git merge develop' from the 'master' branch did not work correctly by adding the variadic_seq_to_seq.hpp file to 'master' that is in my local 'develop'. My understanding of merging in Git is obviously still deficient.

On 26 February 2014 08:12, Edward Diener
Yes, how did you figure that out ?
I went to https://github.com/boostorg/ and looked at what had been recently changed.
there are a lot of differences between master and your merge base, when you merge git won't take these into account - it only deals with changes made after the last merge.
I thought Git was always smart enough to find the merge base and then do a three way merge between the merge base, the commit being merged to and the commit being merged from.
The three way merge works by looking at the differences between the merge base and the two heads. Just considering this file, in 'develop' the file is unchanged. In 'master' the file is in the merge base, but not in master, so git assumes the file has been removed - remember it only looks at those three revisions, and none others. Since there's no change on the develop branch, but there is on the master branch, git picks the master branch. Git doesn't normally make mistakes like this when the merge history is usually more accurate. Although there are some rare edge cases .
This is the tricky thing about transitioning from subversion to git, they have different models of merging, subversion is based on tracking individual changesets, git is based on the history graph. You could try finding the old changesets that haven't been merged and cherry pick them.
This would theoretically mean that I would have to go one by one through every 'develop' change. Ugh ! I thought Git was supposed to save me from having to do all that tedious work.
It typically does, but in this case it doesn't have a decent merge history as that's really difficult to extract from our complicated and messy subversion merge history. You don't have to go through every change, you can look at the differences between master and develop, and then use 'git log' and 'git annotate' to find which revision they come from. Although that gets a lot harder when there's a mix of merged and unmerged changes.
But even after doing that there are still a lot of old changes that need to be merged, mainly in the documentation:
[snip]
You could try to track down all the appropriate changes, but IMO the best thing to do is to get develop to a release-worthy point and just overwrite master from develop.
I could do that. If I did do I just push 'develop' to 'origin/master' ?
I could also, after having done my local merge, just do a 'git diff master develop' and then update master with appropriate changes.
Don't just push develop to origin/master as that will break history. There actually used to be a command for doing this kind of merge, but the git developers decided it was a bad idea and removed it. Here's the way I emulate it: # Make sure we're up to date and on the right branch git fetch git checkout master # Cheap way to tell git that this is a merge: git merge --no-commit -s ours origin/develop # Copy changes over from develop: git diff --binary origin/develop | git apply -R --index # Commit the "merge" git commit I got that technique from http://stackoverflow.com/a/5211321/2434 Once that's done you should be able to merge normally in the future.

On 2/26/2014 3:42 AM, Daniel James wrote:
On 26 February 2014 08:12, Edward Diener
wrote: Yes, how did you figure that out ?
I went to https://github.com/boostorg/ and looked at what had been recently changed.
there are a lot of differences between master and your merge base, when you merge git won't take these into account - it only deals with changes made after the last merge.
I thought Git was always smart enough to find the merge base and then do a three way merge between the merge base, the commit being merged to and the commit being merged from.
The three way merge works by looking at the differences between the merge base and the two heads. Just considering this file, in 'develop' the file is unchanged. In 'master' the file is in the merge base, but not in master, so git assumes the file has been removed - remember it only looks at those three revisions, and none others. Since there's no change on the develop branch, but there is on the master branch, git picks the master branch.
Just for the sake of argument: 'develop' has file 'master' does not have file 'merge base' has file I tell git to merge from 'develop' to 'master'. Why would not git see that a file which is in 'develop' and is in the 'merge base' but is not in 'master', needs to be added to 'master' as part of the three way merge ?
Git doesn't normally make mistakes like this when the merge history is usually more accurate. Although there are some rare edge cases .
It just seems like incorrect 3-way merging logic to me.
This is the tricky thing about transitioning from subversion to git, they have different models of merging, subversion is based on tracking individual changesets, git is based on the history graph. You could try finding the old changesets that haven't been merged and cherry pick them.
This would theoretically mean that I would have to go one by one through every 'develop' change. Ugh ! I thought Git was supposed to save me from having to do all that tedious work.
It typically does, but in this case it doesn't have a decent merge history as that's really difficult to extract from our complicated and messy subversion merge history. You don't have to go through every change, you can look at the differences between master and develop, and then use 'git log' and 'git annotate' to find which revision they come from. Although that gets a lot harder when there's a mix of merged and unmerged changes.
But even after doing that there are still a lot of old changes that need to be merged, mainly in the documentation:
[snip]
You could try to track down all the appropriate changes, but IMO the best thing to do is to get develop to a release-worthy point and just overwrite master from develop.
I could do that. If I did do I just push 'develop' to 'origin/master' ?
I could also, after having done my local merge, just do a 'git diff master develop' and then update master with appropriate changes.
Don't just push develop to origin/master as that will break history.
You are right. I forgot about the history.
There actually used to be a command for doing this kind of merge, but the git developers decided it was a bad idea and removed it. Here's the way I emulate it:
# Make sure we're up to date and on the right branch git fetch git checkout master
# Cheap way to tell git that this is a merge: git merge --no-commit -s ours origin/develop
# Copy changes over from develop: git diff --binary origin/develop | git apply -R --index
# Commit the "merge" git commit
I got that technique from http://stackoverflow.com/a/5211321/2434
Once that's done you should be able to merge normally in the future.
Thanks ! I need to do more reading about 'merge'.

On 26 February 2014 14:23, Edward Diener
Just for the sake of argument:
'develop' has file 'master' does not have file 'merge base' has file
I tell git to merge from 'develop' to 'master'.
Why would not git see that a file which is in 'develop' and is in the 'merge base' but is not in 'master', needs to be added to 'master' as part of the three way merge ?
Imagine 'master' and 'develop' are identical. Then you remove the file in 'master'. This would result in exactly the same situation - the file would be in 'develop' and the merge base, but not in 'master. And after a merge, it'd generally be expected that the file would remain removed. That's the logic behind the merge.
Thanks ! I need to do more reading about 'merge'.
Don't worry about this too much, this situation only occurred because of the way the git modules were created. It should be rare in normal use, only really happening if you do complicated things. It's unfortunate that this is a lot of Boost maintainers' first experience with git.
participants (3)
-
Daniel James
-
Edward Diener
-
Jonathan Wakely