Regression testing Clang/LLVM with VC14.
I have started a new thread as the other one "Testing of clang emualting the VC++ preprocessor on Windows" has become rather long and interlaced with less relevant musings. Here I would like to concentrate on the subject in the title. Sergey suggested I'd run regression tests, in the "other thread". I was hesitant at first, mostly due to the luke warm reception. Thanks to Google's simple coding of their spam filter, I actually missed the post by Niall Douglas of 4th of April. When it was referenced later and after I dug it from the archives, I became encouraged by the fact that maybe I wasn't all wrong and somebody within the community was actually thinking along similar lines as me. Following this I'd decided to take on regression testing. I chose to first simply test boost-develop against VC14, just to see how it goes. Some good, some bad. Was easy to set up (git already included with VC14, all what I needed to do was install python, which I normally have, but I'm having a new laptop). The bad, space required 44GB (advertised as 5GB). Possibly .pdb files (debugger symbols) are filling up disk-space uselessly. Next bad, the process finished with the following lines: # Done writing "C:\boost_develop\results\degski.xml". # Compressing "C:\boost_develop\results\degski.xml"... # Done writing "C:\boost_develop\results\degski.zip". # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Retrying (5 more attempts). # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Retrying (4 more attempts). # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Retrying (3 more attempts). # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Retrying (2 more attempts). # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Retrying (1 more attempts). # Uploading log archive "C:\boost_develop\results\degski.zip" to develop # <lambda> failed with message "[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions" # Giving up. Traceback (most recent call last): File "run.py", line 71, in <module> runner(root) File "C:\boost_develop\boost_regression_src\regression.py", line 287, in __init__ self.main() File "C:\boost_develop\boost_regression_src\regression.py", line 624, in main getattr(self,action_m)() File "C:\boost_develop\boost_regression_src\regression.py", line 591, in command_regression self.command_upload_logs() File "C:\boost_develop\boost_regression_src\regression.py", line 538, in command_upload_logs lambda: File "C:\boost_develop\boost_regression_src\regression.py", line 663, in retry return f() File "C:\boost_develop\boost_regression_src\regression.py", line 546, in <lambda> self.dart_server ) File "C:\boost_develop\boost_regression_src\collect_and_upload_logs.py", line 430, in upload_logs upload_to_ftp( tag, logs_archive, ftp_proxy, debug_level, ftp_url ) File "C:\boost_develop\boost_regression_src\collect_and_upload_logs.py", line 256, in upload_to_ftp ftp = ftplib.FTP( ftp_site ) File "C:\Python27\lib\ftplib.py", line 120, in __init__ self.connect(host) File "C:\Python27\lib\ftplib.py", line 135, in connect self.sock = socket.create_connection((self.host, self.port), self.timeout) File "C:\Python27\lib\socket.py", line 575, in create_connection raise err socket.error: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions I've re-checked this page http://www.boost.org/development/running_regression_tests.htmlagain, there's no mention of "registering" or something similar. Some tests must have failed, got prompted for starting the debugger twice (is the latter normal). This was a test-run (testing the testing), with VC14U2. Assuming the above can get sorted, I would in reality like to test Clang/LLVM (with VC14U2). I would like to ask (Niall Douglas in particular, I guess) what the suggested parameters to run.py should be for this combo. Should I target clang.exe or clang-cl.exe, develop or master. What about Clang/C2? How to reduce disk-space usage? degski
On 9 Apr 2016 at 8:32, degski wrote:
Sergey suggested I'd run regression tests, in the "other thread". I was hesitant at first, mostly due to the luke warm reception.
It's only apparently a luke warm reception due to the volume from those with various axes to grind about winclang's historical decisions. I very much doubt if the majority here isn't very excited about an officially Microsoft supported clang built into Visual Studio, and the fact we finally can write fully standards conforming C++ 14 (sans preprocessor) for the MSVC target. All it requires is someone to raise the target on the Boost regression testing infrastructure and that should bring eyeballs to any compatibility problems. I thank you for investing the effort to make this happen.
This was a test-run (testing the testing), with VC14U2. Assuming the above can get sorted, I would in reality like to test Clang/LLVM (with VC14U2). I would like to ask (Niall Douglas in particular, I guess) what the suggested parameters to run.py should be for this combo.
I am afraid I don't know. I have ... fairly eccentric views regarding Boost in that I'd like my libraries into Boost, but to otherwise have no mandatory dependencies on Boost whatsoever. To that end, I make very little use of Boost anything, especially recently with the post peer review AFIO v2 rewrite which only requires Bjarne's GSL as a mandatory dependency. (gsl-lite is actually the variant I use, Microsoft GSL is a bit heavy. span<T> is amazing, AFIO v2's design is utterly based on span<T> and wouldn't be possible without).
Should I target clang.exe or clang-cl.exe, develop or master. What about Clang/C2?
I can say much more useful about this. I would test BOTH because they are quite different in behaviour once you get into using them in anger. As a general hand waving sweeping generalisation, LLVM clang 3.8 targeting MSVC is the superior tool relative to C2 clang 3.7. It's more stable, it generates better code, it doesn't ICE anything like as much. It does not, however, generate much useful debug info, only line numbers whereas C2 clang generates mostly accurate debug info. I wouldn't bother with LLVM clang before 3.8 for the MSVC target. Previous versions ICEd a lot when you have RTTI and exceptions turned on. Finally, be aware that --analyze works perfectly on winclang, and it will find quite a few problems that Microsoft's /analyze does not.
How to reduce disk-space usage?
Look into the /Z7 debug flag for MSVC. Generates only line number debug info, it's vastly smaller than /Zi. If Boost is turning on /ZI (capital i), then it ought to change to /Zi (small i) as the Edit and Continue feature is excessive on disc space. Finally passing pch=off to b2.exe might help reduce disk space, but slow builds. Good luck and best speed with your efforts. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 4/9/2016 3:19 AM, Niall Douglas wrote:
On 9 Apr 2016 at 8:32, degski wrote:
Sergey suggested I'd run regression tests, in the "other thread". I was hesitant at first, mostly due to the luke warm reception.
It's only apparently a luke warm reception due to the volume from those with various axes to grind about winclang's historical decisions.
There are no "axes to grind" by me, just a reality: winclang's emulation of the VC++ preprocessor, as demonstrated when encoded into Boost PP, is a failure. I wish it were a complete success but it is not. This also means that Boost libraries which use Boost PP will likely fail when compiled with winclang. To paraphrase the old standby: "what is it about this that you do not understand ?" If there is a version of VC++14 that uses clang, and that clang is not emulating the non-standard VC++ preprocessor but is producing the normal standard C++ preprocessor, and that version can be identified via preprocessor defines differently from the clang version that does emulate the non-standard VC++ preprocessor, I would be glad to change Boost PP to support that specific version. Otherwise asking Boost PP to support the clang that emulates the non-standard VC++ preprocessor, and is a failure in doing so, is not happening. snipped...
[Niall Douglas]
the fact we finally can write fully standards conforming C++ 14 (sans preprocessor) for the MSVC target.
VC 2015 Update 2 doesn't support: * C++98: Two-phase name lookup. * C++98: Dynamic exception specifications (deprecated in C++11, and likely to be removed in a future Standard; we don't expect to ever implement this). * C++11: Expression SFINAE support is Partial. With enough workarounds, what's supported is sufficient for result_of/function. * C++11: C99 preprocessor rules. * C++14: Extended constexpr. * C++14: NSDMIs for aggregates. Our STL is C++14 complete and was current with N4567 (pre-Jax) - with minor exceptions as described by https://blogs.msdn.microsoft.com/vcblog/2016/01/22/vs-2015-update-2s-stl-is-... (e.g. without extended constexpr in the compiler, min()/max()/minmax() for initializer_list can't be constexpr). STL
On 9 Apr 2016 at 16:49, Stephan T. Lavavej wrote:
[Niall Douglas]
the fact we finally can write fully standards conforming C++ 14 (sans preprocessor) for the MSVC target.
VC 2015 Update 2 doesn't support:
* C++98: Two-phase name lookup. * C++98: Dynamic exception specifications (deprecated in C++11, and likely to be removed in a future Standard; we don't expect to ever implement this). * C++11: Expression SFINAE support is Partial. With enough workarounds, what's supported is sufficient for result_of/function. * C++11: C99 preprocessor rules. * C++14: Extended constexpr. * C++14: NSDMIs for aggregates.
Our STL is C++14 complete and was current with N4567 (pre-Jax) - with minor exceptions as described by https://blogs.msdn.microsoft.com/vcblog/2016/01/22/vs-2015-update-2s-stl- is-c17-so-far-feature-complete/ (e.g. without extended constexpr in the compiler, min()/max()/minmax() for initializer_list can't be constexpr).
For me the big wins for using winclang with MSVC target over MSVC are (in order of priority): 1. Sometimes when MSVC gives very unhepful error messages, winclang will tell me what exactly is wrong. Sometimes it's the other way round too, either way quickly switching toolset when the errors or warnings are being ambiguous is a big time saver. 2. I can test right now my generalised constexpr paths in Windows only code rather than typing and hoping I'm right when MSVC eventually implements C++ 14 constexpr. For the rest: two phase lookup and NSDMIs for aggregates is easily worked around; the lacking preprocessor I've only ever really cursed maybe a handful of times in my career; full fat Expression SFINAE really would be very handy to have, but I can make do for now. My single biggest feature request not on the roadmap for VS2017 is the ability to parallel compile code with clang and MSVC simultaneously please with a coalesced errors and warnings view. That would be an *amazing* productivity aid. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
M$ on steroids? https://blogs.msdn.microsoft.com/vcblog/2016/05/04/new-code-optimizer/ Let's see how it compares to Clang/LLVM, interesting stuff... -- (\___/) (+'.'+) (")_(") This is Bunny. Copy and paste Bunny into your signature to help him gain world domination!
On 9 April 2016 at 17:49, Stephan T. Lavavej
the fact we finally can write fully standards conforming C++ 14 (sans
[Niall Douglas] preprocessor) for the MSVC target.
VC 2015 Update 2 doesn't support:
* C++98: Two-phase name lookup. * C++98: Dynamic exception specifications (deprecated in C++11, and likely to be removed in a future Standard; we don't expect to ever implement this). * C++11: Expression SFINAE support is Partial. With enough workarounds, what's supported is sufficient for result_of/function. * C++11: C99 preprocessor rules. * C++14: Extended constexpr. * C++14: NSDMIs for aggregates.
Our STL is C++14 complete and was current with N4567 (pre-Jax) - with minor exceptions as described by https://blogs.msdn.microsoft.com/vcblog/2016/01/22/vs-2015-update-2s-stl-is-... (e.g. without extended constexpr in the compiler, min()/max()/minmax() for initializer_list can't be constexpr)
That would be assuming that what VC 2015 Update 2 claims to support doesn't have bugs. Unfortunately, it does have significantly more bugs than Clang.
On 5/7/2016 2:01 PM, Mathias Gaunard wrote:
On 9 April 2016 at 17:49, Stephan T. Lavavej
wrote: the fact we finally can write fully standards conforming C++ 14 (sans
[Niall Douglas] preprocessor) for the MSVC target.
VC 2015 Update 2 doesn't support:
* C++98: Two-phase name lookup. * C++98: Dynamic exception specifications (deprecated in C++11, and likely to be removed in a future Standard; we don't expect to ever implement this). * C++11: Expression SFINAE support is Partial. With enough workarounds, what's supported is sufficient for result_of/function. * C++11: C99 preprocessor rules. * C++14: Extended constexpr. * C++14: NSDMIs for aggregates.
Our STL is C++14 complete and was current with N4567 (pre-Jax) - with minor exceptions as described by https://blogs.msdn.microsoft.com/vcblog/2016/01/22/vs-2015-update-2s-stl-is-... (e.g. without extended constexpr in the compiler, min()/max()/minmax() for initializer_list can't be constexpr)
That would be assuming that what VC 2015 Update 2 claims to support doesn't have bugs. Unfortunately, it does have significantly more bugs than Clang.
Furthermore if the preprocessor is the Clang VC++ emulated preprocessor a number of libraries will fail regression tests since the emulation is broken as far as Boost PP ( and Boost VMD ) are concerned.
participants (5)
-
degski
-
Edward Diener
-
Mathias Gaunard
-
Niall Douglas
-
Stephan T. Lavavej