big problem with dependency changes

newer
How to make bjam use gcc instead...

Robert Ramey

16 Jun 2014 16 Jun '14

5:23 a.m.

Currently the master branch has a few problems. Here is one typical example. My project uses the current master to provide boost libraries which I'm not working on. One file has the statement #include which fails to compile since boost/utility/enable_if.hpp has been moved to core. This raises a few questions. a) Isn't the master branch always supposed to be working? b) There are a lot of intermediate checkins which break the master. Should these be done an a branch and, once they are believed to be all correct, merged in as a group. This would support the concept a) above. c) will I have to change code in the library I maintain to manually change all occurrences of #include to #include ? d) Why are the tests on the master branch passing? Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/big-problem-with-dependency-changes-tp466... Sent from the Boost - Dev mailing list archive at Nabble.com.

Show replies by date

Eric Niebler

16 Jun 16 Jun

5:32 a.m.

On 6/15/2014 10:23 PM, Robert Ramey wrote:

...

Currently the master branch has a few problems. Here is one typical example.

My project uses the current master to provide boost libraries which I'm not working on. One file has the statement

#include

which fails to compile since boost/utility/enable_if.hpp has been moved to core.

<snip> Did you ./b2 headers? I've had to nuke the boost/ directory and return ./b2 headers numerous times in the past few weeks. Eric

Paul A. Bristow

8:28 a.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Eric Niebler Sent: 16 June 2014 06:32 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

<snip>

Did you ./b2 headers? I've had to nuke the boost/ directory and return ./b2 headers numerous times in the past few weeks.

I'm glad I am not alone in having to nuke the /boost directory. Am I right to be worried about this? It should not happen? Why does it happen? Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 01539 561830

Glen Fernandes

9:22 a.m.

On Mon, Jun 16, 2014 at 1:28 AM, Paul A. Bristow wrote:

...

I'm glad I am not alone in having to nuke the /boost directory.

Am I right to be worried about this? It should not happen? Why does it happen?

Paul

I've had no problems on my non-Windows (Linux, OSX) machines. I regularly have to nuke /boost on Windows to get things to work. Glen

Peter Dimov

10:11 a.m.

Glen Fernandes wrote:

...

On Mon, Jun 16, 2014 at 1:28 AM, Paul A. Bristow wrote:

...
I'm glad I am not alone in having to nuke the /boost directory.

Am I right to be worried about this? It should not happen? Why does it happen?

I've had no problems on my non-Windows (Linux, OSX) machines. I regularly have to nuke /boost on Windows to get things to work.

I'm on Windows and b2 has done a good job so far. I've only had to delete /boost maybe once. I usually don't even bother with "b2 headers" - everything seems to work without it. Go figure. :-)

Bjørn Roald

4:28 p.m.

On 06/16/2014 12:11 PM, Peter Dimov wrote:

...

Glen Fernandes wrote:

...
On Mon, Jun 16, 2014 at 1:28 AM, Paul A. Bristow wrote:

...
I'm glad I am not alone in having to nuke the /boost directory.

Am I right to be worried about this? It should not happen? Why does it > happen?

I've had no problems on my non-Windows (Linux, OSX) machines. I regularly have to nuke /boost on Windows to get things to work.

I'm on Windows and b2 has done a good job so far. I've only had to delete /boost maybe once. I usually don't even bother with "b2 headers" - everything seems to work without it. Go figure. :-)

For those of you that struggle with this when you use b2 on your system, You should get a line at the beginning of the output saying: - symlinks supported : yes If not, b2 is not able to create symbolic links for individual files on your system. There may be a number of reasons for this, but on recent MS Windows systems it is usually related to user privileges, the tool used to create symbolic links require elevated privileges. If b2 are not able to create symbolic links for "b2 headers" targets, it will use hard links or file copies. Both of these alternatives to symbolic links have severe issues that could cause you to end up with stale files in the boost folder, or cause you to edit files that are later overwritten by b2. For enabling b2 use of symbolic links on windows, have a look at the following information from the web (quoting from my posting 2 February on the same issue.): refs: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege-ig... http://msdn.microsoft.com/en-us/library/bb530410.aspx http://superuser.com/questions/148937/windows-server-2008-create-symbolic-li... http://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-ho... Most promising may be: Provide SeCreateSymbolicLinkPrivilege privilege to the user who is running the script. If the user is Administrator on the machine you may need to turn off UAC. Note I have not tested any of this as I have no problems on my systems. So please give feedback to the list if any of this work for you. Thanks. -- Bjørn

Paul A. Bristow

5:42 p.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 16 June 2014 17:28 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

On 06/16/2014 12:11 PM, Peter Dimov wrote:

...
Glen Fernandes wrote:

...
On Mon, Jun 16, 2014 at 1:28 AM, Paul A. Bristow wrote:

...
I'm glad I am not alone in having to nuke the /boost directory.

Am I right to be worried about this? It should not happen? Why does it > happen?

I've had no problems on my non-Windows (Linux, OSX) machines. I regularly have to nuke /boost on Windows to get things to work.

I'm on Windows and b2 has done a good job so far. I've only had to delete /boost maybe once. I usually don't even bother with "b2 headers" - everything seems to work without it. Go figure. :-)

For those of you that struggle with this when you use b2 on your system, You should get a line at the beginning of the output saying:

- symlinks supported : yes

If not, b2 is not able to create symbolic links for individual files on your system. There may be a number of reasons for this, but on recent MS Windows systems it is usually related to user privileges, the tool used to create symbolic links require elevated privileges.

If b2 are not able to create symbolic links for "b2 headers" targets, it will use hard links or file copies. Both of these alternatives to symbolic links have severe issues that could cause you to end up with stale files in the boost folder, or cause you to edit files that are later overwritten by b2.

For enabling b2 use of symbolic links on windows, have a look at the following information from the web (quoting from my posting 2 February on the same issue.):

refs: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege- ignored-on-windows-8

http://msdn.microsoft.com/en-us/library/bb530410.aspx

http://superuser.com/questions/148937/windows-server-2008-create-symbolic- link-updated-security-policy-still-gives-pr

http://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7- home-premium-as-a-regular-user

Most promising may be: Provide SeCreateSymbolicLinkPrivilege privilege to the user who is running the

script.

...

If the user is Administrator on the machine you may need to turn off UAC.

Note I have not tested any of this as I have no problems on my systems. So please give feedback to the list if any of this work for you. Thanks.

Thanks. This is very useful info. I didn't think it mattered if hard or soft links but clearly it does. Dont'we need to change the instructions *always* "run from administrator"? Paul

Peter Dimov

5:55 p.m.

Paul A. Bristow wrote:

...

I didn't think it mattered if hard or soft links but clearly it does.

I'm not using soft links, FWIW. b2 uses hard links and junctions for me, and it works reasonably well. Perhaps I'm not stressing it enough though.

Bjørn Roald

17 Jun 17 Jun

5:15 a.m.

On 06/16/2014 07:55 PM, Peter Dimov wrote:

...

Paul A. Bristow wrote:

...
I didn't think it mattered if hard or soft links but clearly it does.

I'm not using soft links, FWIW. b2 uses hard links and junctions for me,

OK, junctions are used for the directories on Windows if symlinks are not working.

...

and it works reasonably well. Perhaps I'm not stressing it enough though.

perhaps, but it could also be something else. I you do not mind, could you try in a clone for used testing only, or at least not where you have work you have not committed: rm boost/filesystem.h b2 headers cd libs/filesystem git reset --hard HEAD then edit boost/filesystem.h or libs/filesystem/include/boost/filesystem.h Then compare them, if they are the same, then I am very interested in understanding why. -- Bjørn

Peter Dimov

8:46 a.m.

Bjørn Roald wrote:

...

I you do not mind, could you try in a clone for used testing only, or at least not where you have work you have not committed:

rm boost/filesystem.h

b2 headers

cd libs/filesystem

git reset --hard HEAD

then edit

boost/filesystem.h or libs/filesystem/include/boost/filesystem.h

Then compare them, if they are the same, then I am very interested in understanding why.

The files are different after I edit one of the two hard links. When I edit in boost/ - which I never do - when I run the tests, b2 then sees the file as modified, but does not update the link in boost/ (the link being newer than the source in include/.) When I edit in include/ - which is what I do - b2 sees the file as modified and updates the link in boost/ to point to it. Moral of the story: don't edit the files in boost/. The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.

Bjørn Roald

9:29 p.m.

On 06/17/2014 10:46 AM, Peter Dimov wrote:

...

Bjørn Roald wrote:

...
I you do not mind, could you try in a clone for used testing only, or at least not where you have work you have not committed:

rm boost/filesystem.h

b2 headers

cd libs/filesystem

git reset --hard HEAD

then edit

boost/filesystem.h or libs/filesystem/include/boost/filesystem.h

Then compare them, if they are the same, then I am very interested in understanding why.

The files are different after I edit one of the two hard links.

OK, that is what I would expect. Thank you for testing,

...

When I edit in boost/ - which I never do - when I run the tests, b2 then sees the file as modified, but does not update the link in boost/ (the link being newer than the source in include/.)

When I edit in include/ - which is what I do - b2 sees the file as modified and updates the link in boost/ to point to it.

Moral of the story: don't edit the files in boost/.

The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit. -- Bjørn

Gavin Lambert

11:23 p.m.

On 18/06/2014 09:29, Bjørn Roald wrote:

...

On 06/17/2014 10:46 AM, Peter Dimov wrote:

...
When I edit in boost/ - which I never do - when I run the tests, b2 then sees the file as modified, but does not update the link in boost/ (the link being newer than the source in include/.)

When I edit in include/ - which is what I do - b2 sees the file as modified and updates the link in boost/ to point to it.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

Perhaps b2 could check both timestamps are equal and sync bidirectionally? Copied or hardlinked files should have the same timestamp (not sure about symlinks, but they should probably be ignored anyway since they won't have this issue), and both manually edited or freshly git-updated files should have a newer timestamp than their counterpart (modulo possible 2 second filesystem resolution). The only thing I can envisage possibly tripping this up is if someone untars/unzips a fresh release build over the top of their existing tree, which could potentially have older timestamps than the counterpart files, if the user has been editing them. (Especially if they're using the same release build as previous, to try to revert to originals.) But that seems like a case in which users ought to have cleaned the tree first in any event, and shouldn't be an issue for upgrades, as those have different dir names embedded in the archive. Although I suspect most users using release builds will not run b2 more than once anyway.

Andrey Semashev

18 Jun 18 Jun

4:55 a.m.

On Wed, Jun 18, 2014 at 1:29 AM, Bjørn Roald wrote:

...

On 06/17/2014 10:46 AM, Peter Dimov wrote:

...
Moral of the story: don't edit the files in boost/.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

Perhaps 'b2 headers' should create these files protected from modification?

Vladimir Prus

19 Jun 19 Jun

5:33 a.m.

On 06/18/2014 08:55 AM, Andrey Semashev wrote:

...

On Wed, Jun 18, 2014 at 1:29 AM, Bjørn Roald wrote:

...
On 06/17/2014 10:46 AM, Peter Dimov wrote:

...
Moral of the story: don't edit the files in boost/.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

Perhaps 'b2 headers' should create these files protected from modification?

That seems like a workable solution to me; I'm not sure whether it will satisfy everybody though. - Volodya

Paul A. Bristow

18 Jun 18 Jun

8:51 a.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 17 June 2014 22:30 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

...

...
Moral of the story: don't edit the files in boost/.

The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

To me, this seems a *big* problem :-( I'd fail the whole idea of using hard or symlinks without some method of protecting the poor unsuspecting hapless users from this elephant trap. Been there, fell in ! Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 01539 561830

Bjørn Roald

4:56 p.m.

On 06/18/2014 10:51 AM, Paul A. Bristow wrote:

...

...
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 17 June 2014 22:30 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

...
...
Moral of the story: don't edit the files in boost/.

The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

To me, this seems a *big* problem :-(

Agree, I for one do not want to down-grade the severity of this.

...

I'd fail the whole idea of using hard or symlinks without some method of protecting the poor unsuspecting hapless users from this elephant trap.

I think symlinks will protect you just fine, and it is supported by b2. So it is not bad at all for users where b2 can use symlinks :-) The problem is to enable symlink support in a simple robust way that does not sacrifice security for users that dont have this by default. On Windows I fail to find a way. I think solving that should be a primary goal now, but it seems we need more detailed understanding of Windows privileges and UAC than I posses. I wonder: Could we possibly write or download an executable that work around this UAC strap-down and create symbolic links for Windows users in the Administrator group that has the SeCreateSymbolicLinkPrivilege set. If that is possible, then a sensible workaround would be feasible. I worry it is not possible, but I will be happy if I am wrong. Down the road there are other options than staging of headers in the legacy monolithic boost folder structure. This may be achieved relatively simply by module level dependencies and multiple -I statements to the compiler. But this approach has its own set of disadvantages and a number of boost tools are not ready for it. There are probably a few pitfalls as well with dubious workarounds, e.g. command line limitations on some systems. End users will likely always be better off with a deployed set of selected boost modules with a single boost header file structure. So that will be simpler with a deployment system that is not using the git repositories. But as some pointed out, they are expected to download, possibly build boost once, then use. So as long as they don't overwrite their copy of an old release, there is no problem. If they do, then it is reasonable to require a b2 clean or similar. -- Bjørn

Gavin Lambert

19 Jun 19 Jun

8:22 a.m.

On 19/06/2014 04:56, Bjørn Roald wrote:

...

The problem is to enable symlink support in a simple robust way that does not sacrifice security for users that dont have this by default. On Windows I fail to find a way. I think solving that should be a primary goal now, but it seems we need more detailed understanding of Windows privileges and UAC than I posses.

What about two-way sync, as I suggested earlier?

...

Could we possibly write or download an executable that work around this UAC strap-down and create symbolic links for Windows users in the Administrator group that has the SeCreateSymbolicLinkPrivilege set. If that is possible, then a sensible workaround would be feasible. I worry it is not possible, but I will be happy if I am wrong.

I think the only way that could work is if a system-level (and thus admin-permissions) service were installed that accepted requests from non-admin users to create symlinks on their behalf. (I hope the security pitfalls of such a thing are obvious to everyone.) It actually gets weirder than that, though. It's possible for a user with administrator rights to enable the symlink creation privilege for all users. After doing that, though, if UAC is still enabled then while any limited user would be able to create symlinks, the admin user will still only be able to do so from an elevated process. (This is because this is one of the privileges that is explicitly denied to the restricted token created for non-elevated-admin-user processes -- and as far as I'm aware there isn't a setting for *that*.) And yes, that's very weird. (Though it's true that symbolic links can be dangerous things, and it's true that most Windows apps aren't expecting them, which can make them even more dangerous. Traversing through symlinks to delete subfolders will ruin many a day.)

...

Down the road there are other options than staging of headers in the legacy monolithic boost folder structure. This may be achieved relatively simply by module level dependencies and multiple -I statements to the compiler. But this approach has its own set of disadvantages and a number of boost tools are not ready for it. There are probably a few pitfalls as well with dubious workarounds, e.g. command line limitations on some systems.

Another possible pitfall is that it may delay detection of cases when different libraries clobber each others' header files (although the naming policies and directory structures should limit this).

Peter Dimov

10:28 a.m.

Gavin Lambert wrote:

...

What about two-way sync, as I suggested earlier?

Two way sync seems somewhat dangerous. Perhaps b2 aborting with an error if it sees a newer file in boost/ will be enough. On second thought, this won't work when the same header is present in more than one module. This happens when a header is being moved and is added to its new module before being removed from its old place. Currently b2 doesn't mind, it just updates the link twice and the later copy wins.

Paul A. Bristow

4:15 p.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 18 June 2014 17:56 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

On 06/18/2014 10:51 AM, Paul A. Bristow wrote:

...
...
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 17 June 2014 22:30 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

...
...
Moral of the story: don't edit the files in boost/.

The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.

Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit.

To me, this seems a *big* problem :-(

Agree, I for one do not want to down-grade the severity of this.

...
I'd fail the whole idea of using hard or symlinks without some method of protecting the poor unsuspecting hapless users from this elephant trap.

I think symlinks will protect you just fine, and it is supported by b2. So it is not bad at all for users where b2 can use symlinks :-)

The problem is to enable symlink support in a simple robust way that does not sacrifice security for users that dont have this by default.

I agree that this is most desirable - or even essential?

...

On Windows I fail to find a way. I think solving that should be a primary goal now, but it seems we need more detailed understanding of Windows privileges and UAC than I posses.

Another complexity I fear is that requiring use of Admin privileges is that it mean users cannot use tools like Tortoise GIT but must use a GIT command line from an elevated privs GIT Bash session. Enforcing command-line only really isn't acceptable, even for library authors. Or is it only running b2 headers that is the problem? Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 01539 561830

Thomas Suckow

7:06 p.m.

...

Or is it only running b2 headers that is the problem?

Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder). I ask, how are people depending on the generated /boost directory? Is it only to build the boost libraries? Regards, Thomas

Vladimir Prus

20 Jun 20 Jun

6:55 a.m.

On 06/19/2014 11:06 PM, Thomas Suckow wrote:

...

...
Or is it only running b2 headers that is the problem?

Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder).

That way, you'd have a command line with 100 -I elements, which is rather inconvenient to look at, or run by hand, and can make Windows unhappy, I think. - Volodya

Bjørn Roald

4:33 p.m.

On 06/20/2014 08:55 AM, Vladimir Prus wrote:

...

On 06/19/2014 11:06 PM, Thomas Suckow wrote:

...
...
Or is it only running b2 headers that is the problem?

Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder).

That way, you'd have a command line with 100 -I elements, which is rather inconvenient to look at, or run by hand, and can make Windows unhappy,

For info: CMake solves this on Windows (at least for linking) by creating a file with commandline options, which can be passed to cl command-line as a single filename option. But this solution is a real pain as these temporary files are gone when you want to see what are passed to the compiler. So there is no simple way to trace what is going on when something is not working. Tracing down some problems in cmake on windows has been a real pain due to this. So, in my opinion, if we can manage to get it right, staging the headers as b2 headers do is so much simpler to reason about. Besides that, the stage structure is probably how end users will see boost headers. So the developers get to eat their own kake, which is a good thing. -- Bjørn

Stephen Kelly

21 Jun 21 Jun

10:59 a.m.

Bjørn Roald wrote:

...

On 06/20/2014 08:55 AM, Vladimir Prus wrote:

...
On 06/19/2014 11:06 PM, Thomas Suckow wrote:

...
...
Or is it only running b2 headers that is the problem?

Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder).

That way, you'd have a command line with 100 -I elements, which is rather inconvenient to look at, or run by hand, and can make Windows unhappy,

For info:

CMake solves this on Windows (at least for linking) by creating a file with commandline options, which can be passed to cl command-line as a single filename option. But this solution is a real pain as these temporary files are gone when you want to see what are passed to the compiler. So there is no simple way to trace what is going on when something is not working. Tracing down some problems in cmake on windows has been a real pain due to this.

FYI: http://cmake.org/gitweb?p=cmake.git;a=blob;f=Modules/Platform/Windows.cmake;... Thanks, Steve.

Bjørn Roald

1:59 p.m.

On 06/21/2014 12:59 PM, Stephen Kelly wrote:

...

Bjørn Roald wrote:

...
On 06/20/2014 08:55 AM, Vladimir Prus wrote:

...
On 06/19/2014 11:06 PM, Thomas Suckow wrote:

...
...
Or is it only running b2 headers that is the problem?

Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder).

That way, you'd have a command line with 100 -I elements, which is rather inconvenient to look at, or run by hand, and can make Windows unhappy,

For info:

CMake solves this on Windows (at least for linking) by creating a file with commandline options, which can be passed to cl command-line as a single filename option. But this solution is a real pain as these temporary files are gone when you want to see what are passed to the compiler. So there is no simple way to trace what is going on when something is not working. Tracing down some problems in cmake on windows has been a real pain due to this.

FYI:

http://cmake.org/gitweb?p=cmake.git;a=blob;f=Modules/Platform/Windows.cmake;...

OK, I see. It would have been nice if it just set CMAKE_START_TEMP_FILE to "" for me when I pass VERBOSE=1 to nmake. But that may re-create the possible issues with the command line length that use of temp files is designed to work around. The best thing to do for build systems using such temp files may be simply to list the content of the file after the command line when doing verbose build output. thanks, -- Bjørn

Bjørn Roald

16 Jun 16 Jun

8:12 p.m.

On 06/16/2014 07:42 PM, Paul A. Bristow wrote:

...

...
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Bjørn Roald Sent: 16 June 2014 17:28 To: boost@lists.boost.org Subject: Re: [boost] big problem with dependency changes

On 06/16/2014 12:11 PM, Peter Dimov wrote:

...
Glen Fernandes wrote:

...
On Mon, Jun 16, 2014 at 1:28 AM, Paul A. Bristow wrote:

...
I'm glad I am not alone in having to nuke the /boost directory.

Am I right to be worried about this? It should not happen? Why does it > happen?

I've had no problems on my non-Windows (Linux, OSX) machines. I regularly have to nuke /boost on Windows to get things to work.

I'm on Windows and b2 has done a good job so far. I've only had to delete /boost maybe once. I usually don't even bother with "b2 headers" - everything seems to work without it. Go figure. :-)

For those of you that struggle with this when you use b2 on your system, You should get a line at the beginning of the output saying:

- symlinks supported : yes

If not, b2 is not able to create symbolic links for individual files on your system. There may be a number of reasons for this, but on recent MS Windows systems it is usually related to user privileges, the tool used to create symbolic links require elevated privileges.

If b2 are not able to create symbolic links for "b2 headers" targets, it will use hard links or file copies. Both of these alternatives to symbolic links have severe issues that could cause you to end up with stale files in the boost folder, or cause you to edit files that are later overwritten by b2.

For enabling b2 use of symbolic links on windows, have a look at the following information from the web (quoting from my posting 2 February on the same issue.):

refs: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege- ignored-on-windows-8

http://msdn.microsoft.com/en-us/library/bb530410.aspx

http://superuser.com/questions/148937/windows-server-2008-create-symbolic- link-updated-security-policy-still-gives-pr

http://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7- home-premium-as-a-regular-user

Most promising may be: Provide SeCreateSymbolicLinkPrivilege privilege to the user who is running the script. If the user is Administrator on the machine you may need to turn off UAC.

Note I have not tested any of this as I have no problems on my systems. So please give feedback to the list if any of this work for you. Thanks.

Thanks. This is very useful info.

I didn't think it mattered if hard or soft links but clearly it does.

Well, it does under certain conditions: A boost/foo/bar.h hard link may end up as a copy of a deleted file that used to be at libs/foo/include/boost/foo/bar.h. A boost/foo/bar.h hard link may also end up as a copy of an old version of a changed file that used to be at libs/foo/include/boost/foo/bar.h. Basically all regular files are hard links to the file content. The hard link in the boost/foo/bar.h and its original are equivalent references to the same file content. They have shared ownership to the file (aka. shared_ptr). If the original is deleted, well, the file lives on as the other hard link lives on. If the original is changed by first deleting, then write to a new file with same name, then the boost/foo/bar.h hard link point to the old version of libs/foo/include/boost/foo/bar.h. Regrettably more tools than I expected do the later type of delete then write sequence rather than in-place overwrite when editing files. Git being one offender :-( Symlinks are just forwarding one name to the target file path. Thus neighter of the above problems causes real issues. You may still have stale symlinks to deleted files, but accessing them would fail when attempted as the original can not be found.

...

Don’t'we need to change the instructions *always* "run from administrator"?

I don't know. Some users are concerned about running as Administrator, and I am not sure I would recommend that at all. Some users may not be allowed to be Administrators in a corporate environment. The problem seems to be that even if you are in the Administrator group, you can not do this as your normal user in scripts. You have to elevate the script, which I assume means you run everything in the script as the Administrator user. That may be a poor idea. Simply setting the SeCreateSymbolicLinkPrivilege for the user seems more promising as general guideline, But there are two issues with that as well: 1. You have to be Administrator to set privelages, or you need help from somebody that is. I would not think this is a problem for most boosters. To change the policies: - Launch secpol.msc via Start or Start → Run. - Open Security Settings → Local Policies → User Rights Assignment. - In the list, find the "Create symbolic links" item, which represents SeCreateSymbolicLinkPrivilege. - Double-click on the item and add yourself (or the whole Users group) to the list. The changes will apply when you log out and log in again. 2. If you are in the Administrators group, you also need to disable User Access Control(UAC). I have no idea if that is a good idea. As this only apply to members of the Administrator group, I guess using a separate user account could be an alternative work around if you or your sysadmin are concerned about turning UAC off. To turn UAC off in Vista/Win7/8/8.1 see here: http://www.howtogeek.com/howto/windows-vista/disable-user-account-control-ua... -- Bjørn

Gavin Lambert

17 Jun 17 Jun

1:04 a.m.

On 17/06/2014 08:12, Bjørn Roald wrote:

...

Regrettably more tools than I expected do the later type of delete then write sequence rather than in-place overwrite when editing files. Git being one offender :-(

It's likely to be defensive programming. Writing to a temporary file with an alternate name and then executing a delete-rename or rename-rename-delete sequence is generally as close as you can get to an atomic update that doesn't leave behind partial files in case of failure (thereby possibly destroying content), at least without "real" transaction support (which does exist but is rarely used in Windows). (This is because writes are slow, but deletes and renames are nearly instant and are less likely to fail.) Thus you'll find version-control systems and other software designed for reliability will do that more often than not, although it's more hit-n-miss with general-purpose applications (eg. editors). And most Windows software is not written to be symlink-aware because they're very rare.

...

If you are in the Administrators group, you also need to disable User Access Control(UAC). I have no idea if that is a good idea. As this only apply to members of the Administrator group, I guess using a separate user account could be an alternative work around if you or your sysadmin are concerned about turning UAC off.

In general, it is not a good idea. While Linux users (and more importantly, applications) are used to the idea of logging in to a completely separate account for day-to-day vs. admin operations, Windows users (and applications) generally expect to use the same account for both. UAC allows the distinction between elevated or not, which lets the non-admin tasks be run more safely (like Linux). While it's possible for a sufficiently disciplined user to run dual accounts like on Linux, this often confuses applications in unexpected ways.

Bjørn Roald

4:38 a.m.

On 06/17/2014 03:04 AM, Gavin Lambert wrote:

...

On 17/06/2014 08:12, Bjørn Roald wrote:

...
Regrettably more tools than I expected do the later type of delete then write sequence rather than in-place overwrite when editing files. Git being one offender :-(

It's likely to be defensive programming. Writing to a temporary file with an alternate name and then executing a delete-rename or rename-rename-delete sequence is generally as close as you can get to an atomic update that doesn't leave behind partial files in case of failure (thereby possibly destroying content), at least without "real" transaction support (which does exist but is rarely used in Windows). (This is because writes are slow, but deletes and renames are nearly instant and are less likely to fail.)

Yes you are right, have done that many times in my own code.

...

Thus you'll find version-control systems and other software designed for reliability will do that more often than not, although it's more hit-n-miss with general-purpose applications (eg. editors).

In fact it was a discussion on the git developer list on the issue, and they decided to ignore the fact that they knew they would break hard links. I think the rationale was partly performance and partly the fact that it would not help much anyway. Other tools would break hardlinks and the reliance on hard links would be broken anyway.

...

And most Windows software is not written to be symlink-aware because they're very rare.

Did you mean hard-link aware? In my mind, whether they are symlink aware or not would not make much of a difference.

...

...
If you are in the Administrators group, you also need to disable User Access Control(UAC). I have no idea if that is a good idea. As this only apply to members of the Administrator group, I guess using a separate user account could be an alternative work around if you or your sysadmin are concerned about turning UAC off.

In general, it is not a good idea. While Linux users (and more importantly, applications) are used to the idea of logging in to a completely separate account for day-to-day vs. admin operations, Windows users (and applications) generally expect to use the same account for both. UAC allows the distinction between elevated or not, which lets the non-admin tasks be run more safely (like Linux). While it's possible for a sufficiently disciplined user to run dual accounts like on Linux, this often confuses applications in unexpected ways.

Yes, I would not like to give the general advice to turn UAC off, what people elect to do is their own business. But this leaves us with a problem for "b2 headers" on Windows, as we have no clear advice of how to deal with lack of symlink support that is simple, easy and secure. At least unless you are happy about not being in the Administrators group. The situation can be improved by making the "b2 header" targets proper targets in b2. For performance reasons, a shortcut is used currently I think. I believe that is also the reason you sometimes need to call "b2 headers" explicitly. Further b2 could detect hardlinks that has become copies in the boost folder. Although this would be an improvement, and it would fix build issues, it is not close to as good as symbolic links. Orphaned hardlinks to deletad header files would still be a potential problem. Also, other tools and the user will still suffer. There is a window from the hardlink relation is broken to next time you build where bad things may happen. E.g. you edit in the wrong file, and then your work is overwritten. Still looking for proper user level symlink support on windows. -- Bjørn

Glen Fernandes

16 Jun 16 Jun

5:33 a.m.

On Sun, Jun 15, 2014 at 10:23 PM, Robert Ramey wrote:

...

My project uses the current master to provide boost libraries which I'm not working on. One file has the statement

#include

which fails to compile since boost/utility/enable_if.hpp has been moved to core.

The code lives in the Core module, but the Core module provides which just forwards to so your code should not fail to compile.

...

c) will I have to change code in the library I maintain to manually change all occurrences of

#include

to

#include

?

No. There's no reason why Core's cannot live on, just marked as deprecated, with all documentation referring to .

...

d) Why are the tests on the master branch passing?

Did you run git submodule update --init? Glen

3812

Age (days ago)

3817

Last active (days ago)

List overview

Download

27 comments

11 participants

participants (11)

Andrey Semashev
Bjørn Roald
Eric Niebler
Gavin Lambert
Glen Fernandes
Paul A. Bristow
Peter Dimov
Robert Ramey
Stephen Kelly
Thomas Suckow
Vladimir Prus